REVIEW 3 major objections 4 minor 39 references
Replacing human Picbreeder users with frontier VLMs yields image archives that differ qualitatively and on metrics from the human baseline; modest selection noise, short context, and many prompted agent personalities measurably shift qualit
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
VLMs can run Picbreeder but produce less refined, more mode-collapsed archives than humans; modest selection noise, short context, and many prompted personalities improve diversity metrics at quality cost.
T0 review reviewed 2026-07-13 challenge →
load-bearing objection First full VLM-agentic Picbreeder recreation with clean ablations; metrics move with noise/memory/agents but risk scoring adversarial junk as open-endedness. the 3 major comments →
In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
When frontier vision-language models fully replace human users inside a faithful Picbreeder loop, the collaborative image archives they produce differ both qualitatively and quantitatively from the historical human baseline. Small amounts of exploratory noise in the selection process increase diversity metrics at some cost to image legibility; surprisingly little history (context length of one prior step) optimizes performance while longer contexts produce pathologies; and increasing the number of distinct agent personalities improves coverage and tree balance without harming semantic recall, yet also propagates nondescript and potentially adversarial high-frequency patterns throughout the a
What carries the argument
The central mechanism is a re-implementation of Picbreeder sessions (archive branching, 20-generation NEAT evolution of CPPN images, publication and rating) driven by VLM agents, evaluated with Semantic Recall over THINGS nouns in SigLIP space, k-covering radii in visual and caption embedding spaces, and the J1 tree-balance index, while three interventions—ε-greedy selection noise, context length CL, and number of LLM-generated personality traits NA—are swept to isolate causal effects.
Load-bearing premise
The chosen metrics of semantic recall, covering radii, and tree balance are assumed to adequately capture the human “x-factor” of bold, refined, meaningful open-ended discovery that the paper sets out to explain.
What would settle it
Blind human raters ranking samples from human versus VLM archives (or from different intervention settings) reverse the ordering given by the paper’s Semantic Recall and coverage metrics; if that occurs, the metrics do not track the intended open-ended quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper faithfully reimplements Picbreeder (CPPN images evolved via NEAT) and replaces human users with frontier VLMs that branch, select, publish, and rate a shared archive. It compares the resulting archives (after 2000 sessions, 6 seeds) against the historical human Picbreeder lineage dataset using Semantic Recall (min SigLIP2 cosine distance summed over THINGS nouns), Visual/Semantic Coverage (k=100 covering radii after farthest-point sampling in embedding spaces), and J1 phylogenetic tree balance. Interventions systematically vary exploratory noise (ε-greedy selection), context length (CL), and number of LLM-generated personality traits (NA). Results show qualitative and quantitative differences from the human baseline; modest ε and short CL improve some metrics while large NA boosts coverage and balance at the cost of high-frequency, potentially adversarial images.
Significance. If the results hold, the work supplies a valuable computational model organism for studying open-ended search without explicit objectives, directly confronting the question of whether current VLMs can sustain the kind of unguided, serendipitous discovery historically associated with human creative and scientific processes. Strengths include a careful reimplementation of Picbreeder mechanics and the published human lineage data, multi-seed quantitative reporting with a random baseline, public code, and explicit ablation of memory, exploration, and multi-agent diversity. These make the paper a useful reference point for subsequent work on agentic open-endedness and quality-diversity search.
major comments (3)
- [§4.1, §5.3, Table 1, Fig. A17] §4.1 and §5.3 (Table 1, Fig. A17): Semantic Coverage and Visual Coverage reach their highest values under NA=1000 precisely when the archive is flooded with high-frequency, uninterpretable grayscale patterns that the authors themselves describe as adversarial hubs. Because the covering-radius metrics treat these patterns as high novelty, the quantitative gains attributed to multi-agent diversity (and to a lesser extent moderate ε) do not necessarily track the human “x-factor” of bold, refined, meaningful discovery that the introduction and §4 set out to explain. A complementary metric that penalizes caption instability or human preference judgments is needed before the causal claims about open-ended ingredients can be accepted.
- [§3.3, Appendix A] §3.3 and Appendix A: Sessions are forcibly terminated after exactly 20 generations with mandatory publication. The authors note that unconstrained agents publish redundantly and that this length is “just shy of the human mean,” yet the constraint removes a core degree of freedom present in the original Picbreeder (variable session length, optional restart/quit). This design choice distances the probe from unconstrained open-ended search and may systematically favor the mode-collapse and refinement behaviors observed under low ε/CL.
- [§4.1] §4.1.1–4.1.3: Semantic Recall relies on a fixed list of 1824 THINGS nouns and SigLIP2 embeddings; Semantic Coverage further depends on VLM-generated captions. Both can be gamed by refined but narrow attractors or by adversarial noise that maps inconsistently under repeated captioning. The paper acknowledges the latter risk but still treats the metrics as primary evidence that interventions recover open-ended potential. Validation against human interestingness ratings on matched archive samples would strengthen the central claim.
minor comments (4)
- [Fig. 2, Figs. A6–A10] Fig. 2 and Fig. A6–A10: Representative grids are useful, but the selection criterion (“seeds with the highest Semantic Recall / Visual Coverage”) should be stated more prominently in the main-text captions so readers can judge cherry-picking risk.
- [Table 1] Table 1: The default setting is highlighted in grey and best-in-sweep in bold, but the absolute best across all conditions is only sometimes green; a single consistent highlighting convention would improve readability.
- [§5, Appendix C] §5 and Appendix C: Semantic Fidelity is introduced as complementary yet is easily gamed by duplicates; either move it to the main metrics suite with an explicit anti-gaming term or de-emphasize it.
- Minor typos and phrasing: “Massachussettes” (author affiliation), “creativiy” (p. 2), occasional missing articles; a light copy-edit pass would suffice.
Circularity Check
Empirical intervention study with external metrics and independent human baseline; no derivation reduces to its own inputs by construction.
full rationale
The paper is an experimental replication of Picbreeder that replaces human users with VLMs, measures archive differences against a fixed historical human dataset, and ablates three design factors (ε-greedy selection noise, context length CL, number of prompted agent personalities NA). Semantic Recall is the sum of minimum SigLIP2 cosine distances from a fixed external list of 1,824 THINGS class names to published images; Visual/Semantic Coverage are k=100 covering radii in independent embedding spaces (SigLIP-2-B-alignet and gemini-embedding-001 after VLM captions); Tree Balance is the J1 index on the phylogenetic tree of publications. None of these quantities is fitted to the VLM archives and then re-predicted. Interventions are independently manipulated and outcomes are reported with means and standard errors over seeds (Table 1, Figs. A6–A10). VLM-generated ratings used only to construct the branching sample are an internal experimental design choice that affects dynamics but do not enter the evaluation metrics or close any definitional loop. Self-citations (e.g., Kumar et al. on CPPN representations) appear only in related-work and appendix analysis of internal weights and are not load-bearing for the main claims about archive quality or intervention effects. There is therefore no self-definitional step, no fitted-input-called-prediction, and no uniqueness or ansatz smuggled via self-citation. Score 0 is the correct outcome for a self-contained empirical paper of this type.
Axiom & Free-Parameter Ledger
free parameters (6)
- session length (generations before forced publish) =
20
- ε (random action probability) =
default 0; best diversity often ~0.25
- context length CL =
default 1
- number of agents / personality traits NA =
default 1; high diversity at 1000
- k for covering radius =
100
- archive sample composition for branching/rating =
100 / 5 categories
axioms (5)
- domain assumption The historical Picbreeder human archive and its lineages are a valid baseline for open-ended image discovery under interactive evolution.
- ad hoc to paper SigLIP2 / gemini embeddings and THINGS class names adequately measure semantic recall and visual/semantic novelty of CPPN images.
- ad hoc to paper LLM-generated second-person personality traits induce meaningfully distinct Picbreeder playstyles without specifying explicit objectives.
- domain assumption NEAT + CPPN image generation with structure/color subnetworks faithfully reimplements the original Picbreeder genotype–phenotype map.
- ad hoc to paper Forced 20-generation sessions and VLM-only branching/rating (even under ε=1) still constitute a valid probe of open-ended search capacity.
invented entities (2)
-
Semantic Recall (min cosine distance sum over THINGS classes in joint embedding space)
no independent evidence
-
Visual / Semantic Coverage via k-covering radius after farthest-point sampling
no independent evidence
Cite this review
Pith. "Pith review of In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models." pith.science (2026). https://pith.science/paper/R7WIXJKR
@misc{pith2026260523908,
author = {Pith},
title = {Pith review of: In Search of the Ingredients of Open-Endedness: Replicating Picbreeder with Large Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/R7WIXJKR}},
note = {Machine review of arXiv:2605.23908}
}
read the original abstract
We are in the midst of large-scale industrial and academic efforts to automate the processes of scientific, technological and creative production through AI-driven assistants. Historically, a fundamental property of these processes in their human form has been their open-endedness: their capacity for generating a seemingly endless supply of novel and meaningful new forms. Do artificial agents have any capacity for such fruitful unguided discovery? To answer this question, we turn to Picbreeder, the canonical exemplar of human-driven open-ended search, in which users collaboratively generated a diverse library of images through interactive evolution of small neural networks. We replicate Picbreeder, replacing human users with frontier Vision Language Models (VLMs). We observe clear qualitative differences between the output of our system and the historical human baseline, and attempt to characterize them using metrics of phylogenetic complexity and visual and semantic salience and novelty. In an effort to identify some of the causal factors contributing these differences, we study the addition of exploratory noise to the agents' selection process, of behavioral diversity between agents, and of narrative momentum in the form of memory of past actions. We make our code available at https://github.com/smearle/picbreeder-vlm.
Figures
Reference graph
Works this paper leans on
-
[1]
Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...
Pith/arXiv arXiv 2025
-
[2]
An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Ja- yaraman, et al. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024
Pith/arXiv arXiv 2024
-
[3]
Quality-diversity through ai feedback, 2023
Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through ai feedback, 2023
2023
-
[4]
Augmenting autotelic agents with large language models
Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc- Alexandre Côté. Augmenting autotelic agents with large language models. In Conference on Lifelong Learning Agents, pages 205–226. PMLR, 2023
2023
-
[5]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025
Pith/arXiv arXiv 2025
-
[6]
Omni-epic: Open- endedness via models of human notions of interestingness with environments programmed in code, 2024
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open- endedness via models of human notions of interestingness with environments programmed in code, 2024
2024
-
[7]
Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. Are quality diversity algorithms better at generating stepping stones than objective-based search? In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 115–116, 2019
2019
-
[8]
Clustering to minimize the maximum intercluster distance
Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical computer science, 38:293–306, 1985
1985
-
[9]
Rachel M Harrison. A comparison of large language model and human perfor- mance on random number generation tasks.arXiv preprint arXiv:2408.09656, 2024
Pith/arXiv arXiv 2024
-
[10]
Things-data, a multimodal collection of large-scale datasets for inves- tigating object representations in human brain and behavior.Elife, 12:e82580, 2023
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, a multimodal collection of large-scale datasets for inves- tigating object representations in human brain and behavior.Elife, 12:e82580, 2023
2023
-
[11]
Agentic pcg: Procedural content generation via tool-using llms, March 2026
Zehua Jiang, Sam Earle, Ahmed Khalifa, and Julian Togelius. Agentic pcg: Procedural content generation via tool-using llms, March 2026. URL https: //doi.org/10.5281/zenodo.19355469
-
[12]
Motif: Intrinsic motivation from artificial intelligence feedback
Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre- Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. 2024
2024
-
[13]
Akarsh Kumar, Jeff Clune, Joel Lehman, and Kenneth O Stanley. Questioning rep- resentational optimism in deep learning: The fractured entangled representation hypothesis.arXiv preprint arXiv:2505.11581, 2025
Pith/arXiv arXiv 2025
-
[14]
Gonzalez, Hao Zhang, and Ion Stoica
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InPro- ceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023
2023
-
[15]
Gemini embedding: Generalizable embeddings from gemini.arXiv preprint arXiv:2503.07891, 2025
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. Gemini embedding: Generalizable embeddings from gemini.arXiv preprint arXiv:2503.07891, 2025
Pith/arXiv arXiv 2025
-
[16]
Evolution through large models
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. InHandbook of Evolutionary Machine Learning, pages 331–366. Springer, 2023
2023
-
[17]
Human planning in stochastic environments
Jordan Lei, Jeroen Olieslagers, Nastaran Arfaei, and Wei Ji Ma. Human planning in stochastic environments
-
[18]
Robust, universal tree balance indices.Systematic biology, 71(5):1210–1224, 2022
Jeanne Lemant, Cécile Le Sueur, Veselin Manojlović, and Robert Noble. Robust, universal tree balance indices.Systematic biology, 71(5):1210–1224, 2022
2022
-
[19]
Rasterfairy
Mario Klingemann. Rasterfairy. URL https://github.com/Quasimondo/ RasterFairy
-
[20]
Miguel, Carolina Feher de Silva, and Marcio Lobo Netto
Alan McIntyre, Matt Kallada, Cesar G. Miguel, Carolina Feher de Silva, and Marcio Lobo Netto. neat-python
-
[21]
Elliot Meyerson, Mark J Nelson, Herbie Bradley, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompt- ing.arXiv preprint arXiv:2302.12170, 2023
arXiv 2023
-
[22]
Aligning machine and human visual representations across abstrac- tion levels.Nature, 623:349–355, 2025
Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Korn- blith, Michael C Mozer, Klaus-Robert Müller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstrac- tion levels.Nature, 623:349–355, 2025
2025
-
[23]
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 427–436, 2015
2015
-
[24]
Understanding innovation engines: Automated creativity and improved stochastic optimization via deep learning
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding innovation engines: Automated creativity and improved stochastic optimization via deep learning. Evolutionary computation, 24(3):545–572, 2016
2016
-
[25]
Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a ty- pology of computational approaches.Frontiers in Neurorobotics, 1, 2007. ISSN 1662-5218. doi: 10.3389/neuro.12.006.2007. URL https://www.frontiersin.org/ articles/10.3389/neuro.12.006.2007
-
[26]
Julien Pourcel, Cédric Colas, Pierre-Yves Oudeyer, and Laetitia Teodorescu. Aces: generating diverse programming puzzles with autotelic language models and semantic descriptors.arXiv preprint arXiv:2310.10692, 2023
arXiv 2023
-
[27]
Picbreeder: evolving pictures collaboratively online
Jimmy Secretan, Nicholas Beato, David B D Ambrosio, Adelein Rodriguez, Adam Campbell, and Kenneth O Stanley. Picbreeder: evolving pictures collaboratively online. InProceedings of the SIGCHI conference on human factors in computing systems, pages 1759–1768, 2008
2008
-
[28]
Picbreeder: A case study in collaborative evolutionary exploration of design space.Evolutionary computation, 19(3):373–403, 2011
Jimmy Secretan, Nicholas Beato, David B D’Ambrosio, Adelein Rodriguez, Adam Campbell, Jeremiah T Folsom-Kovarik, and Kenneth O Stanley. Picbreeder: A case study in collaborative evolutionary exploration of design space.Evolutionary computation, 19(3):373–403, 2011
2011
-
[29]
On creativity and open-endedness.arXiv preprint arXiv:2405.18016, 2024
Lisa Soros, Alyssa Adams, Stefano Kalonaris, Olaf Witkowski, and Christian Guck- elsberger. On creativity and open-endedness.arXiv preprint arXiv:2405.18016, 2024
Pith/arXiv arXiv 2024
-
[30]
Compositional pattern producing networks: A novel abstrac- tion of development.Genetic programming and evolvable machines, 8(2):131–162, 2007
Kenneth O Stanley. Compositional pattern producing networks: A novel abstrac- tion of development.Genetic programming and evolvable machines, 8(2):131–162, 2007
2007
-
[31]
Springer, Switzerland, 2015
Kenneth O Stanley and Joel Lehman.Why Greatness Cannot Be Planned: The Myth of the Objective. Springer, Switzerland, 2015
2015
-
[32]
Evolving neural networks through augmenting topologies.Evolutionary computation, 10(2):99–127, 2002
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies.Evolutionary computation, 10(2):99–127, 2002
2002
-
[33]
Open-endedness: The last grand challenge you’ve never heard of.While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017
Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you’ve never heard of.While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017
2017
-
[34]
On the open-endedness of detecting open-endedness.Artificial Life, 30(3):390–416, 2024
Susan Stepney and Simon Hickinbotham. On the open-endedness of detecting open-endedness.Artificial Life, 30(3):390–416, 2024
2024
-
[35]
MIT press Cambridge, 1998
Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998
1998
-
[36]
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with im- proved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025
Pith/arXiv arXiv 2025
-
[37]
Katherine Van Koevering and Jon Kleinberg. How random is random? evaluating the randomness and humaness of llms’ coin flips.arXiv preprint arXiv:2406.00092, 2024
Pith/arXiv arXiv 2024
-
[38]
On the deleterious effects of a priori objectives on evolution and representation
Brian G Woolley and Kenneth O Stanley. On the deleterious effects of a priori objectives on evolution and representation. InProceedings of the 13th annual conference on Genetic and evolutionary computation, pages 957–964, 2011
2011
-
[39]
fractured, entangled
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open- endedness via models of human notions of interestingness. InThe Twelfth International Conference on Learning Representations, 2024. GECCO ’26, July 13–17, 2026, San Jose, Costa Rica Earle et al. A Limitations & Future Work We could let the agents evolve indefinitely. Let them restart, ...
2024
This paper was first reviewed by grok-4.5 on July 13, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.