Automatic Playtesting for Game Parameter Tuning via Active Learning

Alexander Zook; Eric Fruchter; Mark O. Riedl

REVIEW 3 major objections 5 minor 1 cited by

Automatic Playtesting for Game Parameter Tuning via Active Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Active learning reduces the number of human playtests needed to tune game parameters, for difficulty and control goals alike.

desk verdict Useful case study, but the evaluation metrics are surrogate accuracy (MSE/F1), not the design goals themselves, so the headline claim is one notch ahead of the evidence. read the letter →

arxiv 1908.01417 v1 pith:2BBWLCUT submitted 2019-08-04 cs.AI

Alexander Zook , Eric Fruchter , Mark O. Riedl This is my paper · ORCID

classification cs.AI

keywords ActivelearningmachinegamedesignplaytestingparametertuningGaussianprocessesacquisitionfunctionsshoot-'em-up

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Game designers repeatedly test a game with humans to decide low-level settings such as enemy bullet speed or controller sensitivity, and each round of playtesting costs time and money. This paper argues that many of those tests can be chosen by active learning: instead of sampling parameter settings at random, a model of how settings affect player behavior or preference selects the next setting that is most valuable to test. In a shoot-'em-up case study, active learning beat random sampling for two kinds of design goals—matching a target difficulty by tuning enemy parameters, and choosing controls players prefer from pairwise judgments. The result is a demonstration that the human burden of parameter tuning can be reduced, with the largest gains at small sample sizes and no single acquisition function dominating every case.

What carries the argument

The load-bearing object is the acquisition function, a rule that scores every untried parameter setting by how informative or how promising a playtest there would be, and thereby turns playtesting into a sequential decision process. The paper evaluates four regression acquisition functions (variance, probability of improvement, expected improvement, upper confidence bound) and five classification acquisition functions (entropy, query-by-bagging vote and probability, expected error reduction, variance reduction). These functions carry the argument because the experiments vary only the acquisition function and show that balanced exploration-exploitation (UCB for regression, entropy/QBB for classification) consistently outperforms random selection.

What would settle it

Run a live playtesting loop on a new game with fresh human players each round, letting the acquisition function choose settings online, and compare objective value after N playtests against a random-sampling baseline; if active learning does not beat random sampling around 65-100 playtests for difficulty regression or around 100-200 playtests for control preference classification, the central claim would be falsified.

Watch

Extended reading notes

Core claim

For difficulty-related tuning, treating the number of times a player is hit as a continuous output of three enemy parameters, and training a Gaussian process with the upper-confidence-bound acquisition function, lowered squared error to 203 at 65 playtests and 224 at 280 playtests, compared with 268 and 239 for random sampling. For control-related tuning, treating 'better or worse' control comparisons as a classification output, entropy, query-by-bagging, and expected error reduction improved F1 scores over random sampling for all three classifiers tested; with a Gaussian process classifier, entropy reached 0.763 F1 at 100 samples versus 0.720 random. The paper's conclusion is that active learning is a promising route to reducing the number of playtests needed for both difficulty and control design goals.

Load-bearing premise

The pool-based experiment assumes that a pre-collected set of playtests under random settings faithfully represents what would happen if the acquisition function's chosen settings were given to new players live, and if that correspondence fails, the measured playtest-count savings may not transfer to real tuning loops.

Editorial extensions

If this is right

For difficulty tuning, using upper-confidence-bound selection over a Gaussian process reaches its best performance at about 70 playtests, a level that random sampling never reached on the same data.
The largest gains over random sampling appear at small sample sizes, so active learning is most useful when the playtest budget is limited.
Control settings can be tuned from pairwise 'better/worse' judgments using entropy, query-by-bagging, or expected error reduction, improving F1 over random sampling for Gaussian process, kernel SVM, and neuro-evolved classifiers.
No single acquisition function is best across all tasks and budgets, so playtesting systems need to choose acquisition functions according to the design goal and the number of tests available.
The approach can be embedded in iterative or online design loops to converge more rapidly on good fixed parameters for future use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the pool-based result transfers to live playtesting, the same acquisition functions could tune subjective qualities such as perceived difficulty, fun, or cognitive load whenever those can be scored as regression or classification labels.
Editorial inference: the relative weakness of pure variance and variance-reduction acquisition functions suggests that in sparse high-dimensional playtest pools, maximising uncertainty is a poor guide; designers should favour functions that balance exploration with exploitation or tolerate noisy labels.
Editorial inference: since classification gains were largest for the weakest baseline model, active learning may be most valuable precisely when the playtest model is poor and the budget is small, not when a strong model already exists.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the new thing here is applying active learning to the practical problem of choosing which game parameters to playtest, and the authors are the first I know to include control tuning. That is a real contribution to the game-AI literature. The empirical core is a clean offline, cross-validated comparison of acquisition functions against random sampling on human playtest data. With 138 and 57 players respectively, the gains at small sample sizes are consistent enough that I believe active learning can learn a better surrogate model with fewer playtests. The paper's stated limitations (flat parameters, simple shooter) are honest.

Now the soft spot, and it's the main one: the experiments measure the wrong quantity for the headline claim. Section 5.2 explicitly redefines the classification objective as "maximizing prediction quality (objective function: F1 score)", and all reported results are held-out MSE/F1 of the predictive models. Those are surrogate-model accuracies, not measures of whether the selected parameters achieve the design goals (six hits per wave, controls players prefer). A model with lower MSE or higher F1 does not necessarily lead to a better recommended parameter setting. So the paper demonstrates label-efficient surrogate modeling, not label-efficient parameter tuning. This is not fatal to the paper's practical value, but it means the abstract's "reduce playtesting needed to choose the optimal set of game parameters" is not directly supported by the data. The authors need to either reframe the claim or add an evaluation that scores the final recommended parameters against the design objective.

The other issues are minor in comparison. The pool-based setup assumes that pre-collected random playtests stand in for a live acquisition loop; that transfer is plausible but untested. There are no confidence intervals or significance tests, and no code or data. The citation pattern is fine, not self-citation heavy, and the related work on offline design tools and online adaptation is appropriate.

Who is this for? People working on game analytics, automated playtesting, and maybe studios doing parameter tuning. A reader who wants a careful benchmark of active learning on real playtest data gets something; a reader who wants a validated claim about reducing human playtests in practice should wait for a revision.

Recommendation: send it to peer review. A serious referee can push for the direct design-objective evaluation and clarify the offline-to-online gap. With that fixed, it's a solid applied paper; as is, the headline is one notch ahead of the evidence.

Referee Report

3 major / 5 minor

Summary. The paper proposes casting game parameter tuning as an active learning (AL) problem, in which an acquisition function selects which parameter settings to playtest next so as to reduce the number of human playtests needed to reach a design goal. The authors define two classes of design objectives: a regression objective for tuning enemy parameters so that players are hit exactly six times per wave, and a classification objective for tuning control parameters according to player preference comparisons. They collect human playtest data from an online shoot-'em-up game and evaluate several regression and classification acquisition functions through 10-fold cross-validated, pool-based simulations, comparing against random sampling. The reported results show lower predictive MSE and higher predictive F1 for most acquisition functions compared with random sampling, and the authors conclude that AL is a promising approach for reducing the number of playtests required to achieve a design goal.

Significance. If the claims were fully supported, the paper would make a useful contribution to AI-assisted game design: it identifies a concrete, low-level playtesting task, uses genuine human playtest data, compares a broad set of acquisition functions, and addresses a domain—control tuning—that has received little attention in game AI. The cross-validated pool-based benchmarking against random sampling is a reasonable and standard way to assess label efficiency, and the paper is generally clearly written. However, the significance is currently contingent on closing a substantial gap between the stated design objectives and the metrics actually reported; the experiments demonstrate label-efficient surrogate modeling more directly than they demonstrate more efficient parameter tuning.

major comments (3)

[§5.2, Tables 1–2, Figures 2–5] The experiments evaluate predictive model accuracy, not achievement of the stated design objectives. Section 4 defines the regression design goal as parameters that yield exactly six hits per wave, yet Table 1 and Figures 2–3 report GP MSE on held-out hit counts, which measures how well the model predicts player behavior rather than how close the selected parameter settings are to the design target. For classification, Section 5.2 explicitly redefines the goal as 'maximizing prediction quality (objective function: F1 score)' instead of the Section 4 goal of identifying control settings that players prefer. As a result, the paper demonstrates that AL can train surrogate models with fewer labels, but it does not demonstrate that AL reduces the number of playtests needed to achieve a design goal. I ask the authors to add an evaluation of the final recommended parameter settings under the original design objectives—for regression, report the distance from the chosen parameter setting to the six-hit target using held-out data; for classification, report whether the settings selected by the acquisition functions are actually preferred by players, not just whether pairwise preference labels are predicted accurately.
[§6.1 and Table 1 footnote] The claim that 'AL methods led to significant improvements over the random baseline in all reported results tables' is not supported by any statistical analysis. The paper reports point estimates of MSE and F1 for selected sample sizes without confidence intervals, standard errors, or significance tests, and the figures show bands whose overlap suggests that some differences may be within noise. I request repeated cross-validation with confidence intervals, or paired statistical tests across folds, so that the reader can judge whether the reported improvements are reliable.
[§5.2 and §6.3] The pool-based evaluation assumes that selecting parameter settings from a pre-collected pool of random playtests accurately represents a live playtesting loop in which the acquisition function's chosen settings are given to new human players. The paper does not report whether the pool adequately covers regions near the design optimum, nor whether the acquisition function can actually propose settings in those regions, nor whether player responses change when settings are chosen adaptively. I ask the authors to add coverage diagnostics—for example, the number of pool points near the target, the distribution of acquisition-selected settings versus the random pool—or a small live validation, so that the transfer of the reported reductions in playtest count to a real playtesting process can be assessed.

minor comments (5)

[Title page] The affiliation line contains a typo: 'T echnology' should be 'Technology'.
[§5.1] The paper states that 47 of 57 players provided only binary responses and that the analysis was limited to this subset, but it does not explain how the 'neither' and 'no difference' options were treated when present, nor why players who used those options were excluded. A brief clarification of the preprocessing of preference labels would improve reproducibility.
[Tables 1–2 and Figures 2–5] The figure captions describe bands as 'values that were averaged to produce Table 1' and Table 2, but the bands in Figures 2 and 4 appear to be shaded regions whose meaning (e.g., standard error, range across folds) is not defined. Please state explicitly what the bands represent.
[§6.2] The text says 'Entropy was only effective with few samples,' but Table 2 shows the same GP entropy F1 of 0.763 at both 100 and 200 samples, while the random baseline improves from 0.720 to 0.773 over that range. The narrative could be clarified to distinguish absolute performance from improvement over random sampling.
[References] Reference [26] is listed as 'forthcoming, 2013'; if the paper has since appeared, the full citation should be updated. Also, the statement in Section 6.1 about 'significant improvements' should either be removed or accompanied by the statistical test details described in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the active-learning comparisons are held-out evaluations against a random baseline, and the paper's central claim does not reduce to its inputs by construction.

full rationale

The paper's derivation chain is an empirical comparison: a pool of human playtest data is collected, models are trained on a small random seed, acquisition functions select additional inputs without seeing their outputs, and objective performance is then measured on held-out test data. There is no step in which a fitted parameter is renamed as a prediction or in which a design goal is defined in terms of the quantity being reported. The regression experiments minimize squared error to a target hit rate of 6 and report MSE on held-out data (Section 5.2, Table 1), so the reported values are genuine out-of-sample model errors, not artifacts of fitting. The classification experiments report F1 on held-out preference judgments (Section 5.2, Table 2), again a standard cross-validated evaluation. The paper's self-citations (e.g., [26], [28]) appear only in related-work positioning and are not load-bearing for the experimental results, which use publicly standard AL methods with references to Settles [19] and Bayesian experimental design [3, 5]. The offline pool-based design is an external-validity assumption about transfer to a live playtesting loop, not a circularity, because the acquisition function truly selects inputs from the pool without knowing their labels. One legitimate concern is that the classification 'design goal' of player preference is operationalized in Section 5.2 as maximizing F1, which is a surrogate model-accuracy objective rather than the stated design objective of finding preferred control settings; however, this is a measurement-validity or correctness issue, not an identity between input and output. F1 is computed from labels not used in selecting those labels and is not equivalent by construction to the design goal. Therefore the paper merits a circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted free parameters. It relies on assumptions about the representativeness of the pre-collected pool, the validity of the difficulty proxy, standard ML model assumptions, and the transferability of pool-based AL to live playtesting.

assumptions (4)

domain assumption The pool of pre-collected random playtests is representative of the parameter settings an online AL loop would offer.
Section 5.2 uses the collected data as a pool from which AL selects; transfer to live playtesting assumes this pool spans the relevant parameter space and that adaptive selection does not change player behavior.
domain assumption A player being hit six times per wave is a valid proxy for a difficulty design goal.
Section 4 sets the objective function to six hits per wave; this is a designer's choice, not derived from external criteria.
standard math Standard Gaussian process, KSVM, and neuro-evolution model assumptions hold for gameplay data.
Section 5.2 trains these models without justification; they inherit standard statistical assumptions.
domain assumption Acquisition function behavior transfers from the pool-based setting to a sequential human playtesting process.
The paper evaluates AL by picking from known pool samples; live playtesting would require generating novel parameter settings, which is not covered by the experiment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatic Playtesting for Game Parameter Tuning via Active Learning." pith.science (2026). https://pith.science/paper/2BBWLCUT

@misc{pith2026190801417,
  author       = {Pith},
  title        = {Pith review of: Automatic Playtesting for Game Parameter Tuning via Active Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BBWLCUT}},
  note         = {Machine review of arXiv:1908.01417}
}

read the original abstract

Game designers use human playtesting to gather feedback about game design elements when iteratively improving a game. Playtesting, however, is expensive: human testers must be recruited, playtest results must be aggregated and interpreted, and changes to game designs must be extrapolated from these results. Can automated methods reduce this expense? We show how active learning techniques can formalize and automate a subset of playtesting goals. Specifically, we focus on the low-level parameter tuning required to balance a game once the mechanics have been chosen. Through a case study on a shoot-`em-up game we demonstrate the efficacy of active learning to reduce the amount of playtesting needed to choose the optimal set of game parameters for two classes of (formal) design objectives. This work opens the potential for additional methods to reduce the human burden of performing playtesting for a variety of relevant design concerns.

Figures

Figures reproduced from arXiv: 1908.01417 by the authors.

**Figure 2.** GP performance using different acquisition functions. Shows MSE with an increasing pool of AL-selected training samples. Lower values indicate better performance. Bands indicate values that were averaged to produce [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 4.** Classification performance with different combinations of classifiers and acquisition functions. Higher values indicate better performance. Shows F1 score with an increasing pool of ALselected training samples. Bands indicate values that were averaged to produce [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

NTRL: Encounter Generation via Reinforcement Learning for Dynamic Difficulty Adjustment in Dungeons and Dragons
cs.AI 2025-06 conditional novelty 5.0 of 10

A contextual-bandit RL agent, trained in a D&D combat simulator, generates encounters that lengthen fights, lower party HP, and keep win rates near 80%, though these outcomes are direct reward components rather than i...

Reference graph

Works this paper leans on

45 extracted references · 44 canonical work pages · cited by 1 Pith paper

[1]

the single most important activity a designer engages in

INTRODUCTION Iterative game design practices emphasize the centrality of playtesting to improve and reﬁne a game’s design. Human playtesters provide valuable feedback on audience reactions to a game. Playtesting is often claimed to be“the single most important activity a designer engages in” [7]. Test data in- forms designers of how real players may react...

work page
[2]

Formulating eﬃcient playtesting as an AL problem

work page
[3]

Deﬁning several common playtesting goals in terms of AL metrics

work page
[4]

Unlike prior work on dynamic diﬃculty and adaptive games we focus on the case of decid- ing on a ﬁxed design for future use

Demonstrating the eﬃcacy of AL to reduce the number of playtests needed to optimize (1) diﬃculty-related and (2) control-related game parameters in a case study of a shoot-‘em-up game We believe machine–driven playtesting is a novel use of machine learning in games. Unlike prior work on dynamic diﬃculty and adaptive games we focus on the case of decid- in...

arXiv 1908
[5]

playability,

RELATED WORK Two research areas are closely related to machine playtest- ing: oﬄine game design tools and online game adaptation. Oﬄine game design tools help designers explore possible game designs by deﬁning a high-level space of games through a design language. Online game adaptation changes game designs in real-time based on player actions or game sta...

work page
[6]

For enemy parameter tuning (a regression problem) we found acquisition functions that balance exploration and ex- ploitation (especially UCB) have the best performance

RESULTS AND DISCUSSION Overall our results show AL is a promising approach for reducing the number of playtests needed to achieve a design goal. For enemy parameter tuning (a regression problem) we found acquisition functions that balance exploration and ex- ploitation (especially UCB) have the best performance. For control tuning (a classiﬁcation problem...

work page
[7]

AL Figure 5: Classiﬁcation performance improvement over random sampling with diﬀerent combinations of classiﬁers and acquisition functions

CONCLUSIONS We have shown how playtesting for low-level design pa- rameter tuning can be automated using active learning. AL Figure 5: Classiﬁcation performance improvement over random sampling with diﬀerent combinations of classiﬁers and acquisition functions. Higher val- ues indicate better performance. Shows gains in F1 score with an increasing pool of...

work page
[8]

Active learning provides a generic set of techniques to per- form the playtesting process of choosing a set of design pa- rameters to test toward achieving a design goal

PLAYTESTING AS ACTIVE LEARNING Our goal is to automate mundane playtesting tasks by eﬃ- ciently choosing game design parameters for players to test. Active learning provides a generic set of techniques to per- form the playtesting process of choosing a set of design pa- rameters to test toward achieving a design goal. Playtesting typically involves trade-...

work page

Show all 45 references

[9]

fric- tion

GAME DOMAIN We sought to assess how well AL could reduce the num- ber of playtests needed to achieve a design goal. To con- duct a case study of machine–driven playtesting we devel- oped a simple shoot-‘em-up game (Figure 1). Shoot-‘em-up games emphasize reﬂexes and pattern re...
[10]

neuro-evolution

EXPERIMENTS Our experiments tested whether AL could reduce the num- ber of human playtests needed to tune design parameters compared to a random sampling approach. Random sam- pling is the standard baseline used to evaluate the eﬃcacy of AL models for improving an objective fu...
[11]

Train the regression or classiﬁcation model on the train- ing data set
[12]

Evaluate the objective function for that model on the testing data set
[13]

1For computational reasons we use a gradient-based opti- mization method for network structure, size, and weights, rather than the more common neuro-evolutionary ap- proaches

Use the acquisition function to pick a new input sam- ple from the training pool (without yet knowing the sample output) to improve the objective function. 1For computational reasons we use a gradient-based opti- mization method for network structure, size, and weights, rather...
[14]

Move the selected sample (including the true output) from the training pool into the training data
[15]

Raﬀerty et al

use AL to recommend sets of useful player metrics to track. Raﬀerty et al. [16] optimize game designs oﬄine to learn the most about player cognition. Machine–driven playtesting complements prior uses of AL for game design by focusing on eﬃciently improving designs for player b...
[16]

We used a maximum of 300 training samples in both regres- sion and classiﬁcation

Return to the ﬁrst step and repeat the process until the maximum number of training samples are used. We used a maximum of 300 training samples in both regres- sion and classiﬁcation
[17]

A. I. Schein and L. H. Ungar. Active learning for logistic regression: an evaluation. Machine Learning, 68(3):235–265, 2007

2007
[18]

Abe and H

N. Abe and H. Mamitsuka. Query learning strategies using boosting and bagging. In International Conference on Machine Learning, 1998

1998
[19]

A. W. Bauer, S. Cooper, and Z. Popovic. Automated redesign of local playspace properties. In 8th International Conference on the Foundations of Digital Games, 2013

2013
[20]

player types

combine a rule-based reasoning approach with simula- tion to generate content for a physics-based game. Simu- lation approaches are valuable when design involves an in- tractably large space of possible parameters to test and can serve as input to optimization techniques. Mode...
[21]

E. Brochu. Interactive Bayesian optimization: learning user preferences for graphics and animation . PhD thesis, University of British Columbia, 2010

2010
[22]

Butler, A

E. Butler, A. M. Smith, Y.-E. Liu, and Z. Popovi ¨A˘G. A mixed-initiative tool for designing level progressions in games. In ACM Symposium on User Interface Software and Technology, 2013

2013
[23]

Chaloner and I

K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science, 10 (3):273–304, 1995

1995
[24]

M. Cook, S. Colton, and J. Gow. Initial results from co-operative co-evolution for automated platformer design. In EvoGames 2012, 2012

2012
[25]

Fullerton, C

T. Fullerton, C. Swain, and S. Hoﬀman. Game design workshop: a playcentric approach to creating innovative games. Morgan Kaufmann, 2008

2008
[26]

Harrison and D

B. Harrison and D. L. Roberts. Analytics-driven dynamic game adaption for player retention in scrabble. In IEEE Conference on Computational Intelligence in Games , pages 1–8. IEEE, 2013

2013
[27]

Hastings, R

E. Hastings, R. K. Guha, and K. Stanley. Automatic content generation in the galactic arms race video game. IEEE Transactions on Computational Intelligence and AI in Games , 1:245–263, 2009

2009
[28]

Horswill and L

I. Horswill and L. Foged. Fast procedural level population with playability constraints. In Proceedings of the Eighth Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment , 2012

2012
[29]

Hunicke and V

R. Hunicke and V. Chapman. AI for dynamic diﬃculty adjustment in games. In AAAI Workshop on Challenges in Game Artiﬁcial Intelligence , 2004

2004
[30]

A. Jaﬀe, A. Miller, E. Andersen, Y.-E. Liu, A. Karlin, and Z. Popovic. Evaluating competitive game balance with restricted play. In Proceedings of the Eighth Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment, 2012

2012
[31]

Liapis, H

A. Liapis, H. P. Martınez, J. Togelius, and G. N. Yannakakis. Adaptive game level creation through rank-based interactive evolution. In IEEE Conference on Computational Intelligence in Games , volume 4, pages 71–78. Springer, 2013

2013
[32]

Magerko, B

B. Magerko, B. Stensrud, and L. Holt. Bringing the schoolhouse inside the box - a tool for engaging, individualized training. In 25th Army Science Conference, 2006

2006
[33]

Normoyle, J

A. Normoyle, J. Drake, M. Likhachev, and A. Safonova. Game-based data capture for player metrics. In 8th AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment , 2012

2012
[34]

A. N. Raﬀerty, M. Zaharia, and T. L. Griﬃths. Optimally designing games for cognitive science research. In 34th Annual Conference of the Cognitive Science Society, pages 280–287, 2012

2012
[35]

Seif El-Nasr

M. Seif El-Nasr. Interaction, narrative, and drama: Creating an adaptive interactive narrative using performance arts theories. Interaction Studies, 8(2):209–240, 2007

2007
[36]

B. Settles. Active learning, volume 6. Morgan & Claypool Publishers, 2012

2012
[37]

Shaker, N

M. Shaker, N. Shaker, and J. Togelius. Evolving playable content for cut the rope through a simulation-based approach. In 9th AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment, 2013

2013
[38]

Shaker, G

N. Shaker, G. N. Yannakakis, and J. Togelius. Crowdsourcing the aesthetics of platform games. IEEE Transactions on Computational Intelligence and AI in Games, 5:276–290, 2013

2013
[39]

A. M. Smith, B. Eric, and Z. Popovic. Quantifying over play: Constraining undesirable solutions in puzzle design. In 8th International Conference on the Foundations of Digital Games , 2013

2013
[40]

Srinivas, A

N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning, 2010

2010
[41]

D. Thue, V. Bulitko, M. Spetch, and E. Wasylishen. Interactive storytelling: A player modelling approach. In 3rd AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment , 2007

2007
[42]

Yannakakis and J

G. Yannakakis and J. Togelius. Experience-driven procedural content generation. IEEE Transactions on Aﬀective Computing, 2(3):147–161, 2011

2011
[43]

Yu and M

H. Yu and M. O. Riedl. Personalized interactive narratives via sequential recommendation of plot points. IEEE Trans. Computational Intelligence and AI in Games , forthcoming, 2013

2013
[44]

Yu and T

H. Yu and T. Trawick. Personalized procedural content generation to minimize frustration and boredom based on ranking algorithm. In 7th AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment, 2011

2011
[45]

Zook and M

A. Zook and M. O. Riedl. A temporal data-driven player model for dynamic diﬃculty adjustment. In 8th AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment , 2012

2012

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

the single most important activity a designer engages in

INTRODUCTION Iterative game design practices emphasize the centrality of playtesting to improve and reﬁne a game’s design. Human playtesters provide valuable feedback on audience reactions to a game. Playtesting is often claimed to be“the single most important activity a designer engages in” [7]. Test data in- forms designers of how real players may react...

work page

[2] [2]

Formulating eﬃcient playtesting as an AL problem

work page

[3] [3]

Deﬁning several common playtesting goals in terms of AL metrics

work page

[4] [4]

Unlike prior work on dynamic diﬃculty and adaptive games we focus on the case of decid- ing on a ﬁxed design for future use

Demonstrating the eﬃcacy of AL to reduce the number of playtests needed to optimize (1) diﬃculty-related and (2) control-related game parameters in a case study of a shoot-‘em-up game We believe machine–driven playtesting is a novel use of machine learning in games. Unlike prior work on dynamic diﬃculty and adaptive games we focus on the case of decid- in...

arXiv 1908

[5] [5]

playability,

RELATED WORK Two research areas are closely related to machine playtest- ing: oﬄine game design tools and online game adaptation. Oﬄine game design tools help designers explore possible game designs by deﬁning a high-level space of games through a design language. Online game adaptation changes game designs in real-time based on player actions or game sta...

work page

[6] [6]

For enemy parameter tuning (a regression problem) we found acquisition functions that balance exploration and ex- ploitation (especially UCB) have the best performance

RESULTS AND DISCUSSION Overall our results show AL is a promising approach for reducing the number of playtests needed to achieve a design goal. For enemy parameter tuning (a regression problem) we found acquisition functions that balance exploration and ex- ploitation (especially UCB) have the best performance. For control tuning (a classiﬁcation problem...

work page

[7] [7]

AL Figure 5: Classiﬁcation performance improvement over random sampling with diﬀerent combinations of classiﬁers and acquisition functions

CONCLUSIONS We have shown how playtesting for low-level design pa- rameter tuning can be automated using active learning. AL Figure 5: Classiﬁcation performance improvement over random sampling with diﬀerent combinations of classiﬁers and acquisition functions. Higher val- ues indicate better performance. Shows gains in F1 score with an increasing pool of...

work page

[8] [8]

Active learning provides a generic set of techniques to per- form the playtesting process of choosing a set of design pa- rameters to test toward achieving a design goal

PLAYTESTING AS ACTIVE LEARNING Our goal is to automate mundane playtesting tasks by eﬃ- ciently choosing game design parameters for players to test. Active learning provides a generic set of techniques to per- form the playtesting process of choosing a set of design pa- rameters to test toward achieving a design goal. Playtesting typically involves trade-...

work page