{"id":"5a92738d-2723-4964-a9d3-a180a79e1dcc","arxiv_id":"2508.10914","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage model adding simulated-play funness to a language-model proposal prior best fits novice-invented games, but the model comparison is undermined by including the observed games in the normalization set and by in-sample weight fitting.","lead":"This paper studies how novices invent new board games by analyzing over 450 human-created grid games and testing whether people mentally simulate play to evaluate their ideas. It reports that a model combining language-based proposal likelihood with a simulated funness score fits the invented games best, but the statistical evidence has a flaw.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MaxEnt likelihood comparison is circular because the normalization set includes the very human-created games being explained, inflating θ and the reported χ² improvement.","rationale":"The reader identified the Intuitive Gamer generalization issue as the weakest assumption, and that is a genuine validity concern. However, the more load-bearing problem is the circularity in the MaxEnt normalization: the authors include the very data being explained in the background set used to compute Z(θ). This is a concrete, checkable methodological error that directly undermines the headline χ² result. The reader is right that REJECT is appropriate, but the decisive reason is the normalization contamination, not only the funness-generalization concern. The paper has independent value: the dataset and descriptive analyses are useful, and the authors are appropriately cautious. The circularity can be fixed by a reanalysis with a clean background set, so the verdict is REJECT as currently supported, not a rejection of the research program. I partially agree with the reader because we identify different weakest assumptions, though we converge on the same verdict.","tokens_in":22966,"tokens_out":1271,"duration_ms":12195,"concrete_test":"Recompute the MaxEnt likelihood comparison with Z(θ) computed over a background set that excludes all 402 human-created games (e.g., the 1000 DSL samples plus the 121 original games only). If the best-fit θ is no longer significantly positive, or the χ² improvement over θ=0 drops below the significance threshold, then the central claim fails as stated.","verdict_should_be":"REJECT","load_bearing_attack":"The central quantitative claim is that the two-stage model with simulated funness (θ=5.7) explains human-created games better than proposal-only (θ=0), with χ²=214.6, p<0.0001. This comparison is compromised by how the MaxEnt normalization Z(θ) is computed. In 'Implementation details', the authors state: 'We include the original 121 games and the subset of people's games that are expressible in Ludax as part of the normalization to cover an approximate space of games.' Because the human-created games are included in the normalization set, the likelihood of the data under the model is inflated: exp(f(gi;θ))/Z(θ) is larger than it would be if Z were computed over a background set that excluded the data. This is a textbook presence-only MaxEnt violation: Phillips et al. (2006), cited by the authors, explicitly require that the background sample not include occurrence localities, precisely to avoid this bias. The effect is load-bearing because the reported advantage of the two-stage model over the proposal-only model could be an artifact of the model being scored against a normalization distribution that contains the target data, especially for high-Usim human games that contribute to both numerator and denominator. The reader's verdict is REJECT, and I agree the result is not currently supported, though the fix is a clean reanalysis rather than a conceptual failure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how novice participants invent new grid-based board games after seeing a small set of examples. The authors propose a two-stage cognitive model in which a proposal distribution (approximated by token log-probabilities of formalized game descriptions under a language model) is combined with a model-based evaluation score (U_sim, derived from simulated play under the Intuitive Gamer model). Using a MaxEnt model with a single weight θ, they report that the two-stage model explains the human-created games substantially better than the proposal-only model (χ²=214.6, p<0.0001, θ̂=5.7), and they present descriptive results about context effects and novelty. The manuscript also includes a follow-up human rating study of the created games and a second round of game generation.","tokens_in":23278,"tokens_out":3329,"duration_ms":35535,"significance":"If the central quantitative claim were sound, the paper would offer a valuable computational toolkit for studying open-ended human innovation: it combines large language models for formalization, a domain-specific language for board games, and a cognitively validated model of novice play. The descriptive analyses of context-dependence and novelty are interesting, and the follow-up human ratings provide a useful external check. However, the headline model-comparison result is compromised by two load-bearing methodological issues: the MaxEnt normalization set includes the very games being explained, and the weight θ is fit and significance-tested on the same data. These issues undermine the abstract's central assertion that generated games are best described by a model incorporating population-level model-based estimates of game quality. The problems are fixable through reanalysis, and the paper's own supplementary material already flags the related generalization concern about U_sim.","major_comments":[{"comment":"The normalization set Z(θ) includes the human-created games being modeled. The Implementation details state: 'We include the original 121 games and the subset of people's games that are expressible in Ludax as part of the normalization to cover an approximate space of games.' Including the target games in the background distribution violates the presence-only MaxEnt assumption (Phillips et al., 2006) and inflates the likelihood of the human data, because high-U_sim human games contribute to both numerator and denominator. This is directly load-bearing for the reported χ²=214.6, p<0.0001 advantage of the two-stage model. The analysis should be rerun with a background set that excludes all human-created games, or with a properly held-out partition.","section":"Model-based likelihood analysis / Implementation details"},{"comment":"The Intuitive Gamer funness coefficients were fit to the original 121 games, and the Supplement concedes that the model 'does not as well-generalize to games that are more distant from the base set of games.' Because many human-created games involve shape completion or logical composition (e.g., plus-sign, L-shape, 2x2-square goals) outside the validated class, U_sim is an unvalidated predictor for exactly the games that most need evaluation. The paper should validate U_sim on the human-created games using the follow-up human funness ratings (from the 'Human evaluation of the games people made' section) or explicitly restrict the central claim to games within the validated class.","section":"Game evaluation using Intuitive Gamer model simulations (U_sim) / Supplement: Games Expressible in the Restricted…"},{"comment":"The model comparison sweeps θ over the same human data used to compute the likelihood and reports χ²=214.6, p<0.0001 for the best-fitting θ. This is not a valid significance test because θ is fit on the same data; the test does not account for the number of θ values examined or for the model selection. A proper evaluation requires cross-validation or an information criterion that penalizes the fitted parameter (e.g., AIC/BIC), or a likelihood-ratio test with the correct degrees of freedom under a null that includes the selection procedure.","section":"Results (Figure 5C)"}],"minor_comments":[{"comment":"The phrase 'We include the original original 121 games' contains a duplicated word; please remove the repetition.","section":"Implementation details"},{"comment":"The text refers to 'Table 4' when presenting examples of games with varying base likelihood, but the actual table appears in the Supplement; please label it consistently (e.g., Supplement Table 4) and adjust all cross-references.","section":"Results"},{"comment":"The abstract says 'over 450 human created games' while the dataset section reports N=484; please align these numbers for consistency.","section":"Abstract / Invented games dataset"},{"comment":"The choice of L1 regularization coefficient λ=0.1 is not justified; the paper sweeps θ but not λ, and the sensitivity of the results to λ is not reported. Please add a brief sensitivity analysis or citation for the chosen value.","section":"Model-based likelihood analysis"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a preliminary work-in-progress, and the core quantitative claim is not currently supported because of the presence-only normalization violation and the invalid significance test. However, both issues are fixable with the data already in hand: recomputing Z with an appropriate background set and using cross-validated model comparison would directly address the main concerns. The follow-up human rating study also offers a natural way to validate U_sim on the broader class of human-created games. I therefore recommend a major revision rather than rejection. The novelty of the computational pipeline is moderate but real; the paper's fit for a cognitive-science journal would be strengthened by resolving the statistical issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a promising empirical setup with some nice descriptive results, but the headline model-comparison result is not trustworthy as reported. They include the human-created games in the MaxEnt normalization set, and they fit θ on the same data they test. That is a presence-only error they should know about, since they cite Phillips et al. (2006), which explicitly warns against it.\n\nWhat is actually new: they built a pipeline that takes free-text game descriptions, translates them into Ludax via LLaMA, scores them under a proposal distribution from a Code-LM, and scores them under the Intuitive Gamer model to get a simulated funness measure. Then they run a MaxEnt comparison. The dataset of 484 invented games is from Collins et al. (2026), but the second-generation 213 games and the human-rating follow-up are new. The descriptive results—recombination of seen mechanics, some genuinely novel shape-completion games, board-size preferences, and the relation between creation time and novelty—are credible and interesting. The follow-up human ratings showing people's games are rated more fun than the original 121 set is a nice external validation. The paper is also unusually honest about being preliminary.\n\nThe soft spots are load-bearing. The central claim is that adding θ·U_sim improves likelihood, χ²=214.6, p<0.0001. That comparison is compromised twice. First, Z(θ) includes the very games being modeled—\"the subset of people's games that are expressible in Ludax as part of the normalization.\" That inflates the probability of those games under the model, since they appear in both numerator and denominator. Second, θ is chosen by sweeping on the same human data used for the likelihood ratio, so the p-value is not a valid significance test. The fix is clear: compute Z over a background set that excludes all human games, and do proper model comparison (e.g., cross-validated or with a held-out split). The result might survive a clean reanalysis, but currently it does not support the claim.\n\nSofter concern: U_sim is computed with coefficients fit to funness ratings of the 121 base games. Using it on shape-completion games and logical compositions of conditions is a domain shift. The supplement concedes the model \"does not as well-generalize to games that are more distant from the base set.\" If U_sim is miscalibrated, θ and the \"more fun\" results become artifacts.\n\nWho this is for: cognitive scientists and game-AI people interested in creativity, recombination, and model-based evaluation. The descriptive and human-rating parts are worth a read now. The model comparison needs reanalysis before I would take it seriously. I would still send this to a serious referee: the empirical setup is novel, the data is real, and the flaws are identifiable and fixable. But the current version should come back with major revision, not acceptance.","headline":"Genuinely useful dataset and pipeline, but the headline MaxEnt model comparison is circular—human games are in the normalization set and θ is fit and tested on the same data—so the central quantitative claim does not hold as reported.","tokens_in":23792,"tokens_out":3360,"would_cite":false,"duration_ms":34149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"People invent games by mentally simulating how fun they would be, not just by recombining what they have seen.","keywords":["game design","creativity","innovation","board games","model-based simulation","maximum entropy modeling","Intuitive Gamer model","Ludax"],"falsifier":"Score $U_{\\text{sim}}$'s predicted funness against actual human funness ratings for a held-out set of shape-completion and multi-condition games; if the correlation collapses for games outside the original class, the positive $\\theta$ becomes an artifact of an invalid metric. Alternatively, run the invention task under time pressure or a concurrent memory load: if the simulated-funness advantage and large positive $\\theta$ disappear while recombination stays constant, the evaluation-stage claim is supported; if games remain just as fun, the account fails.","tokens_in":22763,"feed_emoji":"🎲","tokens_out":7091,"duration_ms":64661,"temperature":0.7,"pith_summary":"The paper asks what cognitive mechanisms let ordinary people invent new rules and problems, using board games as its testbed. It argues that invention has two parts: a proposal stage, in which people draw on games they have recently seen, and an evaluation stage, in which they mentally simulate play to judge whether the new game would be fun. Analyzing over 450 human-created games, the authors find that a model combining proposal probability with a simulated-funness score explains the games people made better than proposal alone (fitted weight $\\theta=5.7$; $\\chi^2=214.6$, $p<0.0001$). If right, everyday creativity is not just combinatorial memory; it also involves running an internal model of the thing being invented.","feed_headline":"Mentally playtesting fun explains how people invent games","feed_subtitle":"A model with simulated-funness scores fits invented games far better than one based only on games they'd seen.","key_machinery":"The load-bearing object is the MaxEnt generative model over Ludax programs, a domain-specific language that expresses two-player, perfect-information grid games as simulable environments. Games are formalized from natural language by a code language model; $P_{\\text{base}}$ is the average token log probability of a game's Ludax program conditioned on the context games; and $U_{\\text{sim}}$ is the funness estimate obtained by simulating play with the Intuitive Gamer model, a fast, goal-directed heuristic model of novice play, and combining readouts of balance, challenge, and length. The single weight $\\theta$ controls how much simulated funness shapes generation, and its fitted value of 5.7 carries the argument. The normalization over the intractable space of games is approximated by sampling 1000 Ludax games plus the 121 original games.","core_discovery":"The central claim is that people's invented games are best described by a two-stage generative process: first propose a game by sampling from a distribution conditioned on the games one has seen, then evaluate the proposal by simulating novice play and keeping games that are predicted to be fun. Concretely, the authors define a MaxEnt scoring function $f(g_i;\\theta) = \\log P_{\\text{base}}(g_i) + \\theta \\cdot U_{\\text{sim}}(g_i)$, where $P_{\\text{base}}$ comes from token probabilities of a language model conditioned on the seen context games and $U_{\\text{sim}}$ is a funness score derived from Intuitive Gamer model simulations (balance, challenge, and length readouts). Fitting $\\theta$ on the human data gives $\\hat{\\theta}=5.7$, and the two-stage model fits significantly better than the proposal-only model ($\\theta=0$), reported as $\\chi^2=214.6$, $p<0.0001$. The paper also reports that human-created games score higher in simulated funness than either the 121 context games or randomly sampled Ludax games, even though they are no more likely under the proposal distribution.","pith_inferences":["Editorial inference: the two-stage account predicts an intervention the authors do not run — taxing working memory or time during invention should lower the simulated funness of produced games without lowering how much they recombine seen examples.","Editorial inference: the fitted weight $\\theta$ can be read as a price of funness in generation; comparing $\\theta$ across different invention domains would test how domain-general this evaluation mechanism is.","Editorial inference: the presence-only design cannot distinguish blocked simulation from a quick 'sounds fun' linguistic heuristic; a no-simulation control condition would tell them apart."],"forward_implications":["If people evaluate proposed games by internal simulation, then invention in other rule-based systems, such as laws, protocols, and organizational rules, may likewise depend on the inventor's ability to simulate outcomes rather than merely recombining known templates.","The fitted $\\theta>0$ implies that everyday inventors preferentially produce games that are fun under novice-style play, which predicts that freshly invented games should be judged more fun than the seed games; the paper's follow-up rating study reports exactly this pattern (median funness 59.8 versus 42).","Because the model works over formalized Ludax programs, the same pipeline can operate at scale on natural language descriptions, allowing open-ended creative datasets to be analyzed computationally instead of requiring hand-coded rule spaces.","The paper's descriptive finding that longer creation time is associated with games that leave the restricted grammar suggests the two-stage account predicts that more deliberation leads to more novel proposals."],"supporting_citations":[{"why":"Supplies the 121-game seed set, the dataset of human-created games, and the Intuitive Gamer model of novice play whose simulated readouts define U_sim.","marker":"Collins et al. (2026)"},{"why":"Provides the MaxEnt presence-only modeling method used to compare the proposal-only and two-stage models despite observing only submitted games.","marker":"Phillips et al. (2006)"},{"why":"Defines the Ludax domain-specific language used to formalize natural-language games into simulable programs.","marker":"Todd et al. (2025)"},{"why":"Provides the LLaMA 3.1 8B model whose token probabilities approximate P_base, and the LLaMA 3.3 70B model used for formalization.","marker":"A. Dubey et al. (2024)"},{"why":"Motivates and supplies the language-to-code translation approach that turns freeform game descriptions into structured world models.","marker":"Wong et al. (2023)"}],"fun_headline_variants":["Simulated fun guides how people invent new games","Mental playtesting shapes game invention","Inventing games means simulating fun first","Propose games, then playtest them in your head","Why game invention needs imagined play, not just recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the Intuitive Gamer funness score, which was fit to ratings of the original 121 games, remains a valid measure of fun for the wider class of games people invented, including shape-completion goals and logical combinations of conditions; the supplement concedes this may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Simulated fun guides how people invent new games","Mental playtesting shapes game invention","Inventing games means simulating fun first","Propose games, then playtest them in your head","Why game invention needs imagined play, not just recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1807,"prompt_tokens":1087,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":651}},"tokens_in":703,"tokens_out":720,"duration_ms":7786,"temperature":1.0,"reasoning_tokens":651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:14:53.022582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score $U_{\\text{sim}}$'s predicted funness against actual human funness ratings for a held-out set of shape-completion and multi-condition games; if the correlation collapses for games outside the original class, the positive $\\theta$ becomes an artifact of an invalid metric. Alternatively, run the invention task under time pressure or a concurrent memory load: if the simulated-funness advantage and large positive $\\theta$ disappear while recombination stays constant, the evaluation-stage claim is supported; if games remain just as fun, the account fails.","supporting_citations":[{"cited_title":", Anderson, R P","cited_arxiv_id":null,"evidence_quote":"Provides the MaxEnt presence-only modeling method used to compare the proposal-only and two-stage models despite observing only submitted games."},{"cited_title":", Padula, A G","cited_arxiv_id":null,"evidence_quote":"Defines the Ludax domain-specific language used to formalize natural-language games into simulable programs."}],"review_version":1}