REVIEW 3 major objections 4 minor 38 references
ZapGPT: Free-form Language Prompting for Simulated Cellular Control
T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Trained on a single prompt—'form a cluster'—simulated cells learn to follow eight unseen instructions, including semantic opposites such as 'scatter apart'.
desk verdict ZapGPT is a neat proof-of-concept for free-form language control of a simple 2D cell simulation, but the headline generalization claim is weakened by using the same vision-language model as both the evolutionary fitness and the test evaluator; the 'drift apart' example shows the grader can be permissive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The P2I (prompt-to-intervention) model maps a sentence embedding to a spatial vector field via a 5x5x64 latent and transposed convolutions; the field, applied as an external force over many timesteps, is the only channel through which language touches the cells. The second load-bearing component is the VLM-D2R evaluator (Mistral-Vision), which converts a rendered final image plus the original prompt into a free-form description and a scalar alignment score; that score is the fitness. The (mu+lambda) evolution strategy is the optimizer that closes the loop, since the simulator is non-differentiable.
What would settle it
Use an independent scoring source—human raters or explicit geometric criteria—on the same eight unseen prompts with the same trained controller, and check agreement with the VLM scores. The paper already contains a warning sign: 'Drift apart from one another' received a VLM score of 0.927 while showing no significant change in average pairwise distance, so if other prompts show similar divergence, the claimed language grounding collapses to grader compliance.
Extended reading notes
Core claim
ZapGPT's central claim is that a decentralized agent collective can be guided by free-form text without engineered rewards or prompt templates. The pipeline is closed loop: a SentenceTransformer encodes the prompt; a convolutional decoder expands that embedding into a low-resolution 2D vector field; the field is interpolated and applied to 50 simulated cells with local repulsion for 500 timesteps; a pretrained vision-language model (Mistral-Vision, called D2R) renders the final image, describes it, and returns a 0-to-1 alignment score; and a (mu+lambda) evolution strategy mutates the prompt-to-intervention weights to maximize that score. The authors trained the system on only the prompt 'for
Load-bearing premise
The argument depends on the vision-language model's alignment score being a genuine measure of whether a final image matches the prompt; since the same model both drives evolution and evaluates the trained system, a permissive grader would make the reported generalization an artifact.
Editorial extensions
If this is right
- Natural language alone can serve as a control layer for decentralized systems, with no engineered fitness function or task-specific reward.
- A controller evolved on one prompt can generalize to unseen instructions, including semantic opposites, without retraining—a form of zero-shot behavioral transfer.
- The vision-language alignment score tracks physically measurable spatial change (average pairwise distance) for most tested prompts, so language-based evaluation can stand in for geometric ground truth.
- The same evolution-plus-VLM loop can in principle be pointed at other non-differentiable simulators or embodied collectives, making language prompting a general interface for goal-directed emergence.
- Language-guided control could extend beyond spatial morphology to transcriptional, metabolic, or bioelectric state spaces, where goals are naturally expressed in human terms.
Reading between the lines
- Because the same VLM is both fitness function and grader, the claimed generalization is only as strong as that grader; an independent test with human raters would confirm whether the alignment scores correspond to perceived behavior.
- The 'drift apart from one another' case—high VLM score but no significant change in pairwise distance—suggests the mapping may sometimes exploit image-level cues (e.g., spread to corners) rather than the literal instruction, and is worth probing with more prompts of that form.
- If the P2I embedding is doing the work, then interpolating between prompt embeddings should produce intermediate spatial behaviors; that is a testable prediction the paper does not run.
- The framework suggests a reciprocal design: if cells could generate language as well as consume it, language models could become a shared interface between humans and living collectives—an idea the authors raise but leave open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ZapGPT, a language-driven control pipeline for simulated cellular collectives. A Prompt-to-Intervention (P2I) model maps a sentence embedding produced by SBERT into a spatial vector field via a convolutional decoder; the field is applied as an external force to 50 simulated agents over 500 timesteps. The rendered final state is then scored by a Mistral-Vision vision-language model (D2R), which returns a scalar alignment score measuring how well the image matches the prompt. P2I weights are evolved using a (µ+λ) evolution strategy with the D2R score as the fitness signal, training only on the single prompt “form a cluster.” The authors report that the evolved model generalizes to eight unseen prompts, including scattering prompts with meanings opposite to the training prompt, based on high D2R alignment scores and, for most prompts, statistically significant changes in average pairwise distance. The paper claims this demonstrates an abstract, compositional mapping between language and spatial dynamics, enabling conceptual inversion beyond the training distribution.
Significance. If the claims hold, the work is a useful step toward using free-form natural language as a control interface for decentralized, agent-based systems without engineered reward functions. The paper has concrete strengths: the approach is simple and reproducible, code and data are provided, multiple random seeds and vector-field resolutions are tested, and the use of an independent behavioral metric (average pairwise distance) partially corroborates the VLM-based training signal. The independent PWD evidence supports clustering and three of four scattering prompts, which is nontrivial and suggests the learned mapping is not purely arbitrary. However, the central generalization claim is weakened by a training/evaluation overlap: the same VLM that provided the fitness signal during evolution is also the sole semantic grader for the test prompts. The paper's own data contain a dissociation between VLM score and objective behavior. These issues are fixable with additional controls and independent evaluation, but they are load-bearing for the paper's main conclusion.
major comments (3)
- [§3.1.6, Algorithm 1, §4.5] The D2R alignment score is used as the fitness function during evolution (Section 3.1.6, Algorithm 1), and the same D2R model is then used to evaluate the unseen test prompts (Section 4.5: “we evaluated each with the same D2R model used during training”). Consequently, high D2R scores on unseen prompts may partly reflect grader permissiveness or a prompt-agnostic bias rather than genuine semantic alignment. The independent PWD metric validates clustering and 3/4 scattering prompts, but it does not cover all prompts and cannot by itself establish the abstract compositional mapping claimed in Section 4.5. I recommend replacing or supplementing the D2R-based test evaluation with a held-out VLM that was never used as a fitness signal, or with objective behavioral metrics defined for every prompt, plus a demonstration that D2R scores on unseen prompts correlate with those independent measures
- [Table 1 vs. Figure 11] The dissociation for “Drift apart from one another” is direct evidence that the VLM grader can award a high alignment score when the objective behavior is absent. Table 1 reports a mean VLM score of 0.927 with p=1.30e−7, yet Figure 11 shows no significant change in average pairwise distance for this prompt. Since the paper's own exception demonstrates that the VLM can be permissive in exactly the scattering case central to the semantic-inversion claim, the high scores for other scattering prompts cannot be interpreted without additional evidence that the VLM scores track dispersal. The manuscript should either explain this dissociation, or qualify the semantic-inversion conclusion and provide an independent measure of dispersion for all scattering prompts.
- [§4.5, Table 1] The statistical analysis in Table 1 tests only whether the mean VLM score exceeds 0.5 (one-sided Wilcoxon test). This does not establish that the score reflects alignment with the specific prompt, because the D2R might assign generally high scores to most rendered multi-agent configurations regardless of wording. To support the generalization claim, the paper needs baselines such as: scores for the same rendered images paired with an unrelated prompt; scores from an untrained/random P2I model; or scores for random vector-field interventions. Without such controls, the reported p-values only show that the scores are above an arbitrary threshold, not that the model responds differentially to prompt semantics.
minor comments (4)
- [§4.1, Figures 4–6] Figures 4, 5, and 6 contain only an asterisk and appear to be placeholder captions; they should either be removed or replaced with the actual training-curve figures.
- [§4.4] The text describes panels (a–h) of Figure 9 but does not explicitly map prompt names to panels; adding this mapping would improve reproducibility.
- [§4.5] The phrase “some quantitative evaluation” in the section title undersells the content; consider renaming to “Quantitative evaluation of unseen prompts” or similar.
- [§5.2] The limitations section notes the black-box nature of the VLM but does not mention that the same VLM was used for both fitness and evaluation. This training/evaluation overlap should be acknowledged as a limitation and discussed.
Circularity Check
Generalization is scored by the same VLM that was the fitness signal; 'Drift apart' shows the grader can be permissive.
-
fitted input called prediction
[Section 3.1.6 / Algorithm 1; Section 4.5 / Table 1]
"This score serves as the fitness signal during training... (Sec 3.1.6). Algorithm 1: 'Query VLM with I and prompt p to get fitness score f... P←top µ models in C by score f.' Section 4.5: 'we ran 30 simulations per test prompt and evaluated each with the same D2R model used during training.' Table 1 reports 'alignment scores significantly above 0.5' as evidence of generalization."
The P2I model is evolved to maximize the D2R alignment score, and the same D2R model is then used as the evaluation metric for unseen prompts. High 'alignment scores' are therefore re-measurements of the exact objective that the evolutionary search optimized, not an independent test of semantic grounding. The paper does include an independent PWD metric, but for scattering it supports only 3/4 prompts; in the 'Drift apart from one another' case the VLM awards 0.927 while PWD shows no significant dispersal, demonstrating that the VLM can be permissive. Thus the central claim of abstract compositional generalization rests partly on the same grader that served as fitness, making the result partially circular by construction.
full rationale
The paper's principal evidence for generalization is the D2R alignment score, yet D2R is explicitly the fitness function used during evolution (Algorithm 1; Section 3.1.6). Testing with the same model (Section 4.5) means high scores are not independent confirmation of semantic grounding. The independent pairwise-distance analysis provides some external support, but it is partial and contradicts the VLM in the 'Drift apart' case, showing the grader can be permissive. This is a clear instance of the evaluation metric overlapping with the optimized objective, though not a fully deterministic reduction because the test prompts were unseen during training. Score 6 reflects the partial circularity: the central generalization claim is heavily dependent on the same VLM that defined the fitness, with only secondary independent validation.
Assumptions & free parameters
free parameters (4)
- P2I model weights =
evolved (not stored in paper)
- Vector field grid resolution =
2x2, 5x5, 10x10
- Evolution hyperparameters =
mu=5, lambda=15, G=50, sigma=0.1, 30 seeds
- Simulation constants =
50 agents, 500x500 canvas, T=500, repulsion force
assumptions (4)
- domain assumption Mistral-Vision alignment scores are valid measures of prompt-image semantic alignment.
- domain assumption SBERT embeddings capture the semantic contrasts (e.g., cluster vs scatter) needed for behavioral differentiation.
- domain assumption The 2D physics simulation with repulsion and an external field is a sufficient substrate for emergent language-guided behavior.
- standard math (mu+lambda) evolution strategy can effectively search the P2I weight space given VLM evaluations.
Cite this review
Pith. "Pith review of ZapGPT: Free-form Language Prompting for Simulated Cellular Control." pith.science (2026). https://pith.science/paper/X3Z2NDFG
@misc{pith2026250910660,
author = {Pith},
title = {Pith review of: ZapGPT: Free-form Language Prompting for Simulated Cellular Control},
year = {2026},
howpublished = {\url{https://pith.science/paper/X3Z2NDFG}},
note = {Machine review of arXiv:2509.10660}
}
read the original abstract
Human language is one of the most expressive tools for conveying intent, yet most artificial or biological systems lack mechanisms to interpret or respond meaningfully to it. Bridging this gap could enable more natural forms of control over complex, decentralized systems. In AI and artificial life, recent work explores how language can specify high-level goals, but most systems still depend on engineered rewards, task-specific supervision, or rigid command sets, limiting generalization to novel instructions. Similar constraints apply in synthetic biology and bioengineering, where the locus of control is often genomic rather than environmental perturbation. A key open question is whether artificial or biological collectives can be guided by free-form natural language alone, without task-specific tuning or carefully designed evaluation metrics. We provide one possible answer here by showing, for the first time, that simple agents' collective behavior can be guided by free-form language prompts: one AI model transforms an imperative prompt into an intervention that is applied to simulated cells; a second AI model scores how well the prompt describes the resulting cellular dynamics; and the former AI model is evolved to improve the scores generated by the latter. Unlike previous work, our method does not require engineered fitness functions or domain-specific prompt design. We show that the evolved system generalizes to unseen prompts without retraining. By treating natural language as a control layer, the system suggests a future in which spoken or written prompts could direct computational, robotic, or biological systems to desired behaviors. This work provides a concrete step toward this vision of AI-biology partnerships, in which language replaces mathematical objective functions, fixed rules, and domain-specific programming.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Self- organization in biological systems
Scott Camazine, Jean-Louis Deneubourg, Nigel Franks, J Sneyd, Eric Bonabeau, and Guy Theraulaz. Self- organization in biological systems. 2001
2001
-
[2]
Open problems in artificial life.Artificial life, 6(4):363–376, 2000
Mark A Bedau, John S McCaskill, Norman H Packard, Steen Rasmussen, Chris Adami, David G Green, Takashi Ikegami, Kunihiko Kaneko, and Thomas S Ray. Open problems in artificial life.Artificial life, 6(4):363–376, 2000
2000
-
[3]
Bioelectric signaling: Reprogrammable circuits underlying embryogenesis, regeneration, and cancer.Cell, 184(8):1971–1989, 2021
Michael Levin. Bioelectric signaling: Reprogrammable circuits underlying embryogenesis, regeneration, and cancer.Cell, 184(8):1971–1989, 2021. 11 APREPRINT- SEPTEMBER16, 2025
1971
-
[4]
Programmable self-assembly in a thousand-robot swarm.Science, 345(6198):795–799, 2014
Michael Rubenstein, Alejandro Cornejo, and Radhika Nagpal. Programmable self-assembly in a thousand-robot swarm.Science, 345(6198):795–799, 2014
2014
-
[5]
Synthetic collective intelligence.Biosystems, 148:47–61, 2016
Ricard Solé, Daniel R Amor, Salva Duran-Nebreda, Núria Conde-Pueyo, Max Carbonell-Ballestero, and Raul Montañez. Synthetic collective intelligence.Biosystems, 148:47–61, 2016
2016
-
[6]
Word2vec to behavior: morphology facilitates the grounding of language in machines
David Matthews, Sam Kriegman, Collin Cappelle, and Josh Bongard. Word2vec to behavior: morphology facilitates the grounding of language in machines. In2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4153–4160. IEEE, 2019
2019
-
[7]
Large language models for robotics: A survey.arXiv preprint arXiv:2311.07226, 2023
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey.arXiv preprint arXiv:2311.07226, 2023
arXiv 2023
-
[8]
Top-down models in biology: explanation and control of complex living systems above the molecular level.Journal of The Royal Society Interface, 13(124):20160555, 2016
Giovanni Pezzulo and Michael Levin. Top-down models in biology: explanation and control of complex living systems above the molecular level.Journal of The Royal Society Interface, 13(124):20160555, 2016
2016
Show all 38 references
-
[9]
A text-guided protein design framework.Nature Machine Intelligence, 7(4):580–591, March 2025
Shengchao Liu, Yanjing Li, Zhuoxinran Li, Anthony Gitter, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Arvind Ramanathan, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework.Nature Machine Intelligence, 7(4):580–591, March 2025
2025
-
[10]
Molformer: Motif-based transformer on 3d heterogeneous molecular graphs
Fang Wu, Dragomir Radev, and Stan Z Li. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5312–5320, 2023
2023
-
[11]
Dreamfusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988, 2022
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion.arXiv preprint arXiv:2209.14988, 2022
2022 arXiv
-
[12]
Automating the search for artificial life with foundation models.arXiv preprint arXiv:2412.17799, 2024
Akarsh Kumar, Chris Lu, Louis Kirsch, Yujin Tang, Kenneth O Stanley, Phillip Isola, and David Ha. Automating the search for artificial life with foundation models.arXiv preprint arXiv:2412.17799, 2024
2024 arXiv
-
[13]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...
2021
-
[14]
Oxford university press, 2009
Melanie Mitchell.Complexity: A guided tour. Oxford university press, 2009
2009
-
[15]
Artificial life: An overview
Christopher G Langton. Artificial life: An overview. 1997
1997
-
[16]
WW Norton & Company, 2011
Terrence W Deacon.Incomplete nature: How mind emerged from matter. WW Norton & Company, 2011
2011
-
[17]
Being emergence vs
Jason Winning and William Bechtel. Being emergence vs. pattern emergence: Complexity, control and goal- directedness in biological systems. InThe Routledge handbook of emergence, pages 134–144. Routledge, 2019
2019
-
[18]
Morphogenetic fields in embryogenesis, regeneration, and cancer: Non-local control of complex patterning.Biosystems, 109(3):243–261, September 2012
Michael Levin. Morphogenetic fields in embryogenesis, regeneration, and cancer: Non-local control of complex patterning.Biosystems, 109(3):243–261, September 2012
2012
-
[19]
Adams and Michael Levin
Dany S. Adams and Michael Levin. Endogenous voltage gradients as mediators of cell-cell communication: strategies for investigating bioelectrical signals during pattern formation.Cell and Tissue Research, 352(1):95–122, February 2012
2012
-
[20]
Elowitz and Stanislas Leibler
Michael B. Elowitz and Stanislas Leibler. A synthetic oscillatory network of transcriptional regulators.Nature, 403(6767):335–338, January 2000
2000
-
[21]
Ewen Cameron, Caleb J
D. Ewen Cameron, Caleb J. Bashor, and James J. Collins. A brief history of synthetic biology.Nature Reviews Microbiology, 12(5):381–390, April 2014
2014
-
[22]
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
2018
-
[23]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...
2019
-
[24]
Language to rewards for robotic skill synthesis.Arxiv preprint arXiv:2306.08647, 2023
Wenhao Yu and Gileadi et al. Language to rewards for robotic skill synthesis.Arxiv preprint arXiv:2306.08647, 2023
2023 arXiv
-
[25]
Do as i can and not as i say: Grounding language in robotic affordances
Michael Ahn, Anthony Brohan, and Noah Brown et al. Do as i can and not as i say: Grounding language in robotic affordances. InarXiv preprint arXiv:2204.01691, 2022
2022 arXiv
-
[26]
Lamda: Language models for dialog applications.arXiv preprint arXiv:2201.08239, 2022
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications.arXiv preprint arXiv:2201.08239, 2022. 12 APREPRINT- SEPTEMBER16, 2025
2022 arXiv
-
[27]
Evaluating large language models trained on code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021
2021 arXiv
-
[28]
Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023
2023
-
[29]
Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023
2023 arXiv
-
[30]
From text to life: On the reciprocal relationship between artificial life and large language models
Eleni Nisioti, Claire Glanois, Elias Najarro, Andrew Dai, Elliot Meyerson, Joachim Winther Pedersen, Laetitia Teodorescu, Conor F Hayes, Shyam Sudhakaran, and Sebastian Risi. From text to life: On the reciprocal relationship between artificial life and large language models. I...
2024
-
[31]
Lifegpt: Topology-agnostic generative pretrained transformer model for cellular automata.arXiv preprint arXiv:2409.12182, 2024
Jaime A Berkovich and Markus J Buehler. Lifegpt: Topology-agnostic generative pretrained transformer model for cellular automata.arXiv preprint arXiv:2409.12182, 2024
2024 arXiv
-
[32]
Le, Patrick Erikson, Yanbo Zhang, Michael Levin, and Josh Bongard
Nam H. Le, Patrick Erikson, Yanbo Zhang, Michael Levin, and Josh Bongard. Giving simulated cells a voice: Evolving prompt-to-intervention models for cellular control, 2025
2025
-
[33]
Sentence-bert: Sentence embeddings using siamese bert-networks
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019
2019
-
[34]
Mistral-small-3.1-24b-instruct-2503
Mistral AI. Mistral-small-3.1-24b-instruct-2503. https://huggingface.co/mistralai/Mistral-Small-3. 1-24B-Instruct-2503, 2024. Accessed: 2025-05-06
2024
-
[35]
Evolution strategies–a comprehensive introduction.Natural computing, 1:3–52, 2002
Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies–a comprehensive introduction.Natural computing, 1:3–52, 2002
2002
-
[36]
Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935, 2019
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935, 2019
1910 arXiv
-
[37]
The symbol grounding problem.Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990
Stevan Harnad. The symbol grounding problem.Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990
1990
-
[38]
Biological autonomy.A philo, 2015
Alvaro Moreno and Matteo Mossio. Biological autonomy.A philo, 2015. 13 APREPRINT- SEPTEMBER16, 2025 Figure 10: Distance change across clustering prompts (30 trials each). Significant reduction confirms successful agent convergence. 14 APREPRINT- SEPTEMBER16, 2025 Figure 11: Di...
2015
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.