pith. sign in

arxiv: 2605.13570 · v1 · pith:GD7XVMZUnew · submitted 2026-05-13 · 💻 cs.AI · cs.LG

Learning Local Constraints for Reinforcement-Learned Content Generators

Debosmita Bhaumik , Julian Togelius , Georgios N. Yannakakis , Ahmed Khalifa This is my paper

Pith reviewed 2026-05-14 18:32 UTC · model grok-4.3

classification 💻 cs.AI cs.LG
keywords procedural content generationreinforcement learningwave function collapselocal constraintsgame level generationpuzzle-platform gameshybrid generators
0
0 comments X

The pith

Constraining an RL content generator with local patterns learned by Wave Function Collapse produces playable and visually satisfying puzzle-platform levels.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to combine two generation methods for game levels. Wave Function Collapse learns local patterns from examples to keep levels visually consistent but cannot easily guarantee global traits such as playability. Reinforcement learning can enforce those global traits through rewards but often produces visually poor results. The authors restrict the RL agent's available actions to only those allowed by the learned local constraints. When the approach works, generators create levels that meet both visual and functional requirements, as shown with some Lode Runner examples after tuning inputs and training details.

Core claim

By constraining the action space of a PCGRL generator with local constraints learned by WFC from existing content, the hybrid method produces levels that satisfy both local visual patterns and global properties such as playability.

What carries the argument

The restricted action space of the PCGRL policy, limited to actions compatible with WFC-learned local constraints.

Load-bearing premise

The local constraints learned by WFC stay compatible with the RL reward signal and do not restrict the policy so much that global properties become unreachable.

What would settle it

A set of training runs in which every policy that stays inside the local constraints fails to reach high global reward values, or in which all output levels either break local patterns or fail playability checks.

Figures

Figures reproduced from arXiv: 2605.13570 by Ahmed Khalifa, Debosmita Bhaumik, Georgios N. Yannakakis, Julian Togelius.

Figure 1
Figure 1. Figure 1: System overview of the WCRL framework data. The model tries to learn the underlying distribution of the training data; afterwards, the trained model is used to gen￾erate new content. Various machine learning approaches have been explored for automated content generation, which range from Markov models [28], LSTM networks [34], Generative Adversarial Networks GANs) [24], [40], AutoEncoders [23], to recent L… view at source ↗
Figure 3
Figure 3. Figure 3: The different input levels for all the experiments where [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Levels generated using single input frames [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Levels generated using multiple input frames. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Levels generated using highly diverse multiple input frames. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Compares the playability of levels generated using [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average number of cells collapsed at every timestep. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Diversity of playable levels generated using different [PITH_FULL_IMAGE:figures/full_fig_p007_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Levels generated by PCGRL agent patterns leads to a decrease in the diversity of the generated levels, due to the smaller number of available patterns. But it helps increasing number of playable levels. We believe that removing rare patterns helped the framework to focus on the most common patterns that usually lead to fully connected levels, rather than having these unique patterns that appear rarely in … view at source ↗
Figure 12
Figure 12. Figure 12: Number of patterns across different experimental [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
read the original abstract

Constraint-based game content generators that learn local constraints from existing content, such as Wave Function Collapse (WFC), can generate visually satisfying game levels but face challenges in guaranteeing global properties, such as playability. On the other hand, reinforcement-learning trained generators can guarantee global properties -- because such properties can easily be included in reward functions -- but the results can be visually dissatisfying. In this paper, we explore ways to combine these methods. Specifically, we constrain the action space of a PCGRL generator with constraints learned by WFC, effectively allowing the PCGRL generator to achieve global properties while forced to adhere to local constraints. To better analyze how this hybrid content generation method operates, we vary the number and type of inputs, and we test whether to randomly collapse the starting state and exclude rare patterns. While the method is sensitive to hyperparameter tuning, the best of our trained generators produce visually satisfying and playable puzzle-platform game levels -- such as Lode Runner levels -- with desired global properties.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes a hybrid procedural content generation method for puzzle-platform games (e.g., Lode Runner) that restricts the action space of a PCGRL reinforcement-learning generator using local constraints extracted by Wave Function Collapse (WFC) from a corpus of existing levels. The approach aims to combine WFC's visual fidelity with PCGRL's ability to optimize global properties such as playability via reward functions. Experiments vary WFC input types, random starting-state collapse, and exclusion of rare patterns; the abstract asserts that the best resulting generators produce visually satisfying, playable levels with desired global properties.

Significance. If the empirical results were supported by quantitative metrics, baselines, and statistical analysis, the hybrid method could meaningfully advance PCG by addressing the complementary weaknesses of pure constraint-based and pure RL generators. The work is empirical rather than theoretical and introduces no new equations or parameter-free derivations.

major comments (2)
  1. [Abstract] Abstract: the central claim that 'the best of our trained generators produce visually satisfying and playable puzzle-platform game levels ... with desired global properties' is unsupported by any reported quantitative metrics, success rates, reward curves, baseline comparisons (unconstrained PCGRL or pure WFC), statistical significance tests, or details on playability measurement and number of evaluated levels.
  2. [Methods] The integration of WFC constraints into the PCGRL action space (described in the methods) is presented without evidence that the restricted feasible set still intersects the high-reward region for global properties; the noted sensitivity to hyperparameters and ad-hoc mitigations (random collapse, rare-pattern exclusion) do not constitute such evidence.
minor comments (1)
  1. [Abstract] The abstract and text would benefit from explicit definition of the reward function components and the precise mechanism by which WFC patterns restrict PCGRL actions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We agree that the current manuscript would be strengthened by additional quantitative evidence and baseline comparisons, and we will revise accordingly to address the points raised.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'the best of our trained generators produce visually satisfying and playable puzzle-platform game levels ... with desired global properties' is unsupported by any reported quantitative metrics, success rates, reward curves, baseline comparisons (unconstrained PCGRL or pure WFC), statistical significance tests, or details on playability measurement and number of evaluated levels.

    Authors: We acknowledge that the abstract's claim would be better supported by quantitative results. The current version relies primarily on qualitative examples of generated levels to demonstrate visual quality and playability. In the revision we will add explicit metrics: playability success rates obtained by running a solver or agent on generated levels, reward curves from training, direct comparisons against unconstrained PCGRL and pure WFC baselines, the number of levels evaluated per configuration, and statistical significance tests. These additions will be placed in a new results subsection and referenced from the abstract. revision: yes

  2. Referee: [Methods] The integration of WFC constraints into the PCGRL action space (described in the methods) is presented without evidence that the restricted feasible set still intersects the high-reward region for global properties; the noted sensitivity to hyperparameters and ad-hoc mitigations (random collapse, rare-pattern exclusion) do not constitute such evidence.

    Authors: We agree that simply noting hyperparameter sensitivity and the use of random collapse or rare-pattern exclusion does not by itself demonstrate that high-reward global properties remain reachable. In the revised manuscript we will include new experiments that directly measure achieved reward values under the constrained action space, compare them to the unconstrained PCGRL baseline, and report the fraction of runs that reach target global properties (e.g., playability). These results will be presented alongside the existing ablation studies on input types and collapse strategies. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical hybrid generator with no derivations

full rationale

The manuscript contains no equations, derivations, or parameter-fitting steps that reduce any claimed result to its own inputs by construction. All reported outcomes are obtained by training PCGRL policies under WFC-derived action-space restrictions and evaluating the generated levels against playability and visual metrics; no self-citation chain, uniqueness theorem, or ansatz is invoked to justify the central feasibility claim. The work therefore remains self-contained against external benchmarks of generated content.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that local patterns extracted by WFC are sufficient to preserve visual quality while still permitting an RL policy to reach global objectives; several hyperparameters are tuned without reported values.

free parameters (2)
  • number and type of WFC inputs
    The paper explicitly varies the number and type of inputs to the constraint learner.
  • hyperparameters for RL training
    The abstract states the method is sensitive to hyperparameter tuning.
axioms (1)
  • domain assumption WFC can reliably extract local constraints from existing game content that remain useful when transferred to an RL generator.
    This is the core premise enabling the hybrid.

pith-pipeline@v0.9.0 · 5482 in / 1240 out tokens · 48537 ms · 2026-05-14T18:32:41.490871+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · 2 internal anchors

  1. [1]

    Hierarchical semantic wave function collapse

    Shaad Alaka and Rafael Bidarra. Hierarchical semantic wave function collapse. InProceedings of the 18th International Conference on the Foundations of Digital Games, 2023

    work page 2023

  2. [2]

    Leveraging reinforcement learn- ing and wavefunctioncollapse for improved procedural level generation

    Mathias Babin and Michael Katchabaw. Leveraging reinforcement learn- ing and wavefunctioncollapse for improved procedural level generation. InProceedings of the 16th International Conference on the Foundations of Digital Games, FDG ’21, 2021

    work page 2021

  3. [3]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    work page 2016

  4. [4]

    Tile-based map generation using wave function col- lapse in ’caves of qud’, 2022

    Brian Bucklew. Tile-based map generation using wave function col- lapse in ’caves of qud’, 2022. https://www.youtube.com/watch?v= AdCgi9E90jw

    work page 2022

  5. [5]

    Sturgeon-graph: Constrained graph generation from examples

    Seth Cooper. Sturgeon-graph: Constrained graph generation from examples. InProceedings of the 18th International Conference on the Foundations of Digital Games, 2023

    work page 2023

  6. [6]

    Procedural level generation with diffusion models from a single example

    Shiqi Dai, Xuanyu Zhu, Naiqi Li, Tao Dai, and Zhi Wang. Procedural level generation with diffusion models from a single example. InPro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10021–10029, 2024

    work page 2024

  7. [7]

    Learning controllable content generators

    Sam Earle, Maria Edwards, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Learning controllable content generators. In2021 IEEE Conference on Games (CoG), 2021

    work page 2021

  8. [8]

    Adversarial reinforcement learning for procedural content generation

    Linus Gissl ´en, Andy Eakins, Camilo Gordillo, Joakim Bergdahl, and Konrad Tollmar. Adversarial reinforcement learning for procedural content generation. In2021 IEEE Conference on Games (CoG), 2021

    work page 2021

  9. [9]

    Wave function collapse, 2016

    Maxim Gumin. Wave function collapse, 2016. https://github.com/ mxgmn/WaveFunctionCollapse

    work page 2016

  10. [10]

    A closer look at invalid action masking in policy gradient algorithms.The International FLAIRS Conference Proceedings, 2022

    Shengyi Huang and Santiago Onta ˜n´on. A closer look at invalid action masking in policy gradient algorithms.The International FLAIRS Conference Proceedings, 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 9

    work page 2022

  11. [11]

    Learning controllable 3d level generators

    Zehua Jiang, Sam Earle, Michael Green, and Julian Togelius. Learning controllable 3d level generators. InProceedings of the 17th International Conference on the Foundations of Digital Games, 2022

    work page 2022

  12. [12]

    Isaac Karth and Adam M. Smith. Wavefunctioncollapse is constraint solving in the wild. InProceedings of the 12th International Conference on the Foundations of Digital Games, 2017

    work page 2017

  13. [13]

    Isaac Karth and Adam M. Smith. Addressing the fundamental tension of pcgml with discriminative learning. InProceedings of the 14th International Conference on the Foundations of Digital Games, 2019

    work page 2019

  14. [14]

    Isaac Karth and Adam M. Smith. Wavefunctioncollapse: Content gen- eration via constraint solving and machine learning.IEEE Transactions on Games, 2022

    work page 2022

  15. [15]

    Pcgrl: procedural content generation via reinforcement learning

    Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pcgrl: procedural content generation via reinforcement learning. In Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE’20, 2020

    work page 2020

  16. [16]

    Automatic generation of game content using a graph- based wave function collapse algorithm

    Hwanhee Kim, Seongtaek Lee, Hyundong Lee, Teasung Hahn, and Shinjin Kang. Automatic generation of game content using a graph- based wave function collapse algorithm. In2019 IEEE Conference on Games (CoG), 2019

    work page 2019

  17. [17]

    miwfc - designer empowerment through mixed-initiative wave function collapse

    Thijmen Stefanus Leendert Langendam and Rafael Bidarra. miwfc - designer empowerment through mixed-initiative wave function collapse. InProceedings of the 17th International Conference on the Foundations of Digital Games, 2022

    work page 2022

  18. [18]

    Tile pattern kl-divergence for analysing and evolving game levels

    Simon M Lucas and Vanessa V olz. Tile pattern kl-divergence for analysing and evolving game levels. InProceedings of the Genetic and Evolutionary Computation Conference, pages 170–178, 2019

    work page 2019

  19. [19]

    Practical pcg through large language models

    Muhammad U Nasir and Julian Togelius. Practical pcg through large language models. In2023 IEEE Conference on Games (CoG), pages 1–4, 2023

    work page 2023

  20. [20]

    Ex- panding wave function collapse with growing grids for procedural map generation

    Tobias Nordvig Møller, Jonas Billeskov, and George Palamas. Ex- panding wave function collapse with growing grids for procedural map generation. InProceedings of the 15th International Conference on the Foundations of Digital Games, 2020

    work page 2020

  21. [21]

    Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Max- imilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8, 2021

    work page 2021

  22. [22]

    Enhancing wave function collapse with design-level constraints

    Arunpreet Sandhu, Zeyuan Chen, and Joshua McCoy. Enhancing wave function collapse with design-level constraints. InProceedings of the 14th International Conference on the Foundations of Digital Games, 2019

    work page 2019

  23. [23]

    Controllable level blending between games using variational autoencoders.arXiv preprint arXiv:2002.11869, 2020

    Anurag Sarkar, Zhihan Yang, and Seth Cooper. Controllable level blending between games using variational autoencoders.arXiv preprint arXiv:2002.11869, 2020

    work page arXiv 2002

  24. [24]

    Interactive evolution and exploration within latent level-design space of generative adversarial networks

    Jacob Schrum, Jake Gutierrez, Vanessa V olz, Jialin Liu, Simon Lucas, and Sebastian Risi. Interactive evolution and exploration within latent level-design space of generative adversarial networks. InProceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 148–156, 2020

    work page 2020

  25. [25]

    Nelson.Procedural Content Generation in Games: A Textbook and an Overview of Current Research

    Noor Shaker, Julian Togelius, and Mark J. Nelson.Procedural Content Generation in Games: A Textbook and an Overview of Current Research. Springer, 2016

    work page 2016

  26. [26]

    Yannakakis

    Tianye Shu, Jialin Liu, and Georgios N. Yannakakis. Experience-driven pcg via reinforcement learning: A super mario bros study. In2021 IEEE Conference on Games (CoG), 2021

    work page 2021

  27. [27]

    Path of destruction: Learning an iterative level generator using a small dataset

    Matthew Siper, Ahmed Khalifa, and Julian Togelius. Path of destruction: Learning an iterative level generator using a small dataset. In2022 IEEE Symposium Series on Computational Intelligence (SSCI), pages 337–343. IEEE, 2022

    work page 2022

  28. [28]

    Learning to generate video game maps using markov models.IEEE transactions on computational intelligence and AI in games, 9(4):410–422, 2016

    Sam Snodgrass and Santiago Ontan ´on. Learning to generate video game maps using markov models.IEEE transactions on computational intelligence and AI in games, 9(4):410–422, 2016

    work page 2016

  29. [29]

    Multi-domain level generation and blending with sketches via example-driven bsp and variational autoencoders

    Sam Snodgrass and Anurag Sarkar. Multi-domain level generation and blending with sketches via example-driven bsp and variational autoencoders. InFoundations of Digital Games. ACM, 2020

    work page 2020

  30. [30]

    Gen- erating lode runner levels by learning player paths with lstms

    Kynan Sorochan, Jerry Chen, Yakun Yu, and Matthew Guzdial. Gen- erating lode runner levels by learning player paths with lstms. In Proceedings of the 16th International Conference on the Foundations of Digital Games. Association for Computing Machinery, 2021

    work page 2021

  31. [31]

    Wave function collapse in bad north, 2018

    Oskar Stalberg. Wave function collapse in bad north, 2018. https: //www.youtube.com/watch?v=0bcZb-SsnrA

    work page 2018

  32. [32]

    Illuminating the space of beatable lode runner levels produced by various generative adversarial networks

    Kirby Steckel and Jacob Schrum. Illuminating the space of beatable lode runner levels produced by various generative adversarial networks. InGECCO. ACM, 2021

    work page 2021

  33. [33]

    Growing 3d artefacts and functional machines with neural cellular automata.ArXiv, 2021

    Shyam Sudhakaran, Djordje Grbic, Siyan Li, Adam Katona, Elias Najarro, Claire Glanois, and Sebastian Risi. Growing 3d artefacts and functional machines with neural cellular automata.ArXiv, 2021

    work page 2021

  34. [34]

    Super Mario as a String: Platformer Level Generation Via LSTMs

    Adam Summerville and Michael Mateas. Super mario as a string: Platformer level generation via lstms.arXiv preprint arXiv:1603.00930, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  35. [35]

    Procedural content generation via machine learning (pcgml)

    Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmg˚ard, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (pcgml). IEEE Transactions on Games, 10(3):257–270, 2018

    work page 2018

  36. [36]

    The VGLC: The Video Game Level Corpus

    Adam James Summerville, Sam Snodgrass, Michael Mateas, and San- tiago Ontan ´on. The vglc: The video game level corpus.arXiv preprint arXiv:1606.07487, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  37. [37]

    Autoencoder and evolutionary algorithm for level generation in lode runner

    Sarjak Thakkar, Changxing Cao, Lifan Wang, Tae Jong Choi, and Julian Togelius. Autoencoder and evolutionary algorithm for level generation in lode runner. InConference on Games. IEEE, 2019

    work page 2019

  38. [38]

    How townscaper works: A story four games in the making, 2022

    Tommy Thompson. How townscaper works: A story four games in the making, 2022. https://www.youtube.com/watch?v= 1fvJ5sHh6A

    work page 2022

  39. [39]

    Level generation through large language models

    Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. InProceedings of the 18th International Conference on the Foundations of Digital Games, FDG ’23, 2023

    work page 2023

  40. [40]

    Evolving mario levels in the latent space of a deep convolutional generative adversarial network

    Vanessa V olz, Jacob Schrum, Jialin Liu, Simon M Lucas, Adam Smith, and Sebastian Risi. Evolving mario levels in the latent space of a deep convolutional generative adversarial network. InProceedings of the genetic and evolutionary computation conference, pages 221–228, 2018

    work page 2018