{"id":"4a9b4b80-0cbe-4394-be0b-6387b6c096bb","arxiv_id":"2412.17799","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A vision-language foundation model can score simulated videos, enabling automated search for target, open-ended, and diverse artificial life simulations.","lead":"This paper uses image-language AI models like CLIP to automatically search for interesting artificial life simulations across several substrates. The method finds simulations matching text prompts, discovers cellular automata judged open-ended, and maps diverse lifeforms, uncovering new Lenia and Boids patterns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ASAL's open-endedness and discovery claims rely on CLIP embedding novelty as a proxy for human open-endedness, but no human evaluation or open-endedness-specific representation ablation is provided. A human rating study would test whether the metric tracks genuine open-endedness.","rationale":"The reader's weakest assumption—that CLIP/DINOv2 embeddings align with human judgments of interestingness, novelty, and diversity—is the load-bearing pillar of ASAL because every objective (Eq. 2, 3, 4) is defined in that representation. I agree with the reader that this is unvalidated. My analysis sharpens the concern to open-endedness: Eq. (3) uses historical nearest-neighbor novelty in CLIP space over a 2048-timestep rollout, and the paper's only FM ablation (Section 4.5) covers illumination, not open-endedness, and is qualitative. Thus the headline result about open-ended CAs could reflect CLIP-specific invariances rather than genuine open-ended behavior. A human rating study of the discovered CAs is the direct test. I am not raising an internal inconsistency; the paper is transparent about the subjectivity of open-endedness and provides code and detailed appendices. The main issue is external validity, so the paper is best described as a promising method demonstration under conditional acceptance, pending human grounding. The finite-horizon nature of the metric is a secondary aspect of the same concern: a long-run study (e.g., 50,000 steps) would reveal whether novelty saturates, but the human study is the more fundamental check on representation alignment.","tokens_in":17950,"tokens_out":7366,"duration_ms":67005,"concrete_test":"Pre-register a human study: select the 10 highest-scoring open-ended CAs from Figure 5, Conway's Game of Life, and 10 randomly selected CAs; render matched short videos; have at least 20 naive raters rate each video on 'keeps generating new interesting patterns' and overall interestingness; compute Spearman correlation between mean human scores and the Eq. (3) CLIP novelty scores. Significance and positive correlation would support the human-alignment assumption; weak or negative correlation would show the open-endedness claim is an artifact of the chosen embedding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's key claim—discovery of open-ended CAs 'like Conway's Game of Life' (Section 4.2, Figure 5)—is supported entirely by the open-endedness score in Eq. (3): expected historical nearest-neighbor novelty of CLIP embeddings. Section 3.2 explicitly outsources the subjectivity of open-endedness to the representation function, then assumes CLIP approximates human perception, citing prior perceptual-similarity work. However, no experiment in the paper compares Eq. (3) or Eq. (4) to human ratings. Section 4.5 ablates the FM only for illumination (Lenia/Boids) and reports purely qualitative results ('CLIP seems slightly better than DINOv2'), not a quantitative or human-grounded comparison. Thus the top-ranked open-ended CAs might be novel in CLIP space while being repetitive or uninteresting to humans; a CA that cycles between visually distinct but semantically redundant states would score highly. The claim of 'previously unseen lifeforms' is similarly not checked against a systematic inventory of known Lenia/Boids forms. These are validation gaps rather than internal contradictions, but the strongest claims outrun the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ASAL, a framework that uses vision-language foundation models (CLIP, DINOv2) to search for artificial life simulations. Three search mechanisms are proposed: supervised target search via text prompts (Eq. 2), open-endedness search via historical nearest-neighbor novelty in FM embedding space (Eq. 3), and illumination via nearest-neighbor diversity (Eq. 4). The method is applied to Boids, Particle Life, Game of Life/Life-like CA, Lenia, and Neural Cellular Automata. The authors report discovery of target phenomena, previously unseen Lenia and Boids forms, Life-like CAs that are open-ended like Conway's Game of Life, and simulation atlases. Additional FM-based metrics quantify nonlinearity, parameter sensitivity, and plateau behavior.","tokens_in":18330,"tokens_out":3915,"duration_ms":33979,"significance":"If validated, ASAL would be a significant advance: it is substrate-agnostic, requires no hand-crafted objectives beyond a pretrained FM, and covers three distinct search goals. The paper ships code, presents an exhaustive brute-force evaluation of all 262,144 Life-like CAs for the open-endedness objective, and demonstrates qualitative success across diverse substrates. The FM-based quantification in Section 4.4 is a useful contribution. However, the strongest claims—open-endedness and 'previously unseen' lifeforms—are validated only through CLIP embedding statistics and qualitative inspection, not through human judgments or independent benchmarks. The contribution is currently a promising proof of concept rather than a fully established result.","major_comments":[{"comment":"The central claim that ASAL discovers cellular automata that are open-ended 'like Conway's Game of Life' is supported exclusively by the historical CLIP-novelty score in Eq. (3). Since Conway's Game of Life is ranked by that same score (top 5% in Figure 5c), the finding is metric-relative, and no independent validation shows that the score tracks human perceptions of open-endedness or known OE properties such as unbounded growth, glider production, or computational universality. The paper is transparent that the subjectivity is outsourced to the representation, but a human rating study or a comparison with established complexity/open-endedness indicators (e.g., Shannon entropy over time, diversity of emergent attractors) is necessary to justify the wording 'open-ended like Conway's Game of Life'.","section":"Section 3.2, Eq. (3); Section 4.2"},{"comment":"The claims of discovering 'previously unseen Lenia and Boids lifeforms' are not checked against an exhaustive inventory of known forms (e.g., Chan's Lenia expanded universe or LeniaBreeder's catalog), and the supporting evidence consists of qualitative screenshots. In addition, the supervised-target and illumination experiments use a single initial state seed per evaluation (Appendix B.1) and report no quantitative diversity or novelty metrics. To make the novelty claim load-bearing, the authors should either compare against a systematic catalog or provide quantitative evidence that the discovered final states lie outside previously documented behavioral classes.","section":"Section 4.3 and 4.1; Appendix B.1"},{"comment":"The foundation-model ablation for illumination reports no numeric results; the text states only that 'CLIP seems slightly better than DINOv2' and that both are 'qualitatively better' than pixel representations. Given that the paper's core premise is the advantage of FM representations, the ablation should include a quantitative measure of coverage, diversity, or human-rated quality of the illuminated atlas. As written, the ablation does not substantiate the claimed importance of the FM choice.","section":"Section 4.5, Figure 8"},{"comment":"The open-endedness ranking of Life-like CAs depends on several procedural choices—32 subsampled timesteps from a 2,048-step rollout, 256 random initial states with sparsity sampled from U(0.05, 0.4), and a single CLIP ViT variant—but no sensitivity analysis is provided. Since the headline result is a ranking that places Conway's Game of Life only in the top 5%, it is important to show that the ranking is stable under changes in timestep subsampling and initial-state sparsity; otherwise the specific top-ranked rules in Figure 5a may be artifacts of these choices.","section":"Appendix B.2; Section 4.2"}],"minor_comments":[{"comment":"Typo: 'diverse phonenema' should be 'diverse phenomena'.","section":"Section 3.3"},{"comment":"Grammar: 'an \"diversity\" score' should be 'a \"diversity\" score'.","section":"Equation (4)"},{"comment":"The phrase 'colors match the equation below' is not meaningful in a black-and-white print version; please add explicit labels or markers instead.","section":"Section 3.1, Figure 2"},{"comment":"Figure 7a reports CLIP similarity values but provides no baseline or calibration to indicate what counts as high or low similarity; adding a random-pair baseline would help interpretation.","section":"Section 4.4"},{"comment":"The plateau-halting metric in Figure 7d is presented for a single Lenia simulation; a few examples or a quantitative threshold would strengthen the claimed utility as a halting condition.","section":"Section 4.4(d)"},{"comment":"The Lenia substrate description notes that the search space is centered at a solution from Chan (2020), but the anchor and its bounds are not specified; this makes the search space definition incomplete for replication.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong proof-of-concept for a new paradigm, but the top-line claims about open-endedness and novel lifeforms currently outrun the evidence. I would urge the editor to ask for the human-evaluation and sensitivity analyses outlined in the major comments; with those additions the paper could become acceptable. The exhaustive Life-like CA search and the released code are notable strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading and worth refereeing. The core idea—using CLIP and DINOv2 as reward models to search across ALife substrates for target, open-ended, and diverse simulations—is genuinely new in this combination, and the authors have done the engineering work to make it credible. The brute-force search over all 262,144 Life-like CAs, with Conway ranking in the top 5% on their open-endedness score, is a nice sanity check. The released code and detailed appendices make replication straightforward, and the quantitative analyses in Section 4.4 (parameter interpolation, the \"more is different\" scaling, parameter sensitivity, plateau detection) are clever and useful. The paper is also honest: Section 3.2 openly outsources the subjectivity of open-endedness to the representation function.\n\nThe soft spots are exactly where the reader's report and stress-test point: the key claims rest on the assumption that CLIP embedding novelty tracks human interestingness, yet there is no human evaluation anywhere in the paper. The FM ablation in Section 4.5 is qualitative and only for illumination; it doesn't test the open-endedness metric against human judgment. \"Previously unseen\" Lenia and Boids forms are not checked against any systematic inventory, and the supervised-target experiments use a single initial-state seed per evaluation. These are validation gaps, not internal contradictions. The central framework holds up as a proof of concept; what doesn't hold up is the overreach in the abstract's language about \"open-ended like Conway's Game of Life\" and \"previously unseen lifeforms.\" A CA that cycles through visually distinct but semantically redundant states could score highly on their Eq. 3, and that's a real concern for the strongest claim.\n\nThe authors clearly know the field, cite the relevant prior work (LeniaBreeder, OMNI, novelty search, MAP-Elites), and don't hide their proxy assumption. This is a serious paper, not a sloppy one. It deserves to go to peer review, and I would not desk-reject it. The revisions I'd want: a human-rating study comparing CLIP novelty scores to human judgments of interestingness, at least for a sample of the discovered CAs; a check of the \"previously unseen\" claim against known catalogs of Lenia/Boids patterns; and multi-seed evaluations for the supervised targets. With those, the claims could match the evidence. As is, I'd cite it for the framework and the method, not for the open-endedness result.\n\nBring it to reading group; it will generate a good discussion about what counts as open-endedness and whether CLIP is a reliable observer for ALife.","headline":"ASAL is a solid, well-engineered method paper that brings vision-language FMs into ALife search, but its strongest claims about open-endedness and newly discovered lifeforms outrun the validation.","tokens_in":18739,"tokens_out":1728,"would_cite":true,"duration_ms":18417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated Search for Artificial Life (ASAL) claims that vision-language foundation models can automate the discovery of lifelike simulations, finding target phenomena, temporally open-ended novelty, and diverse simulation landscapes…","keywords":["artificial life","foundation models","CLIP","open-endedness","simulation search","cellular automata","Lenia","Boids"],"falsifier":"A blind human-preference study would settle the load-bearing assumption. Take the highest-scoring open-ended cellular automata from the brute-force search and an equal number of randomly chosen automata, render their rollouts, and ask human raters which are more interesting or more lifelike. If preference does not track the ASAL open-endedness score, or if humans find the random automata just as engaging, then the claimed human alignment of the foundation-model representation is not doing the work.","tokens_in":17763,"feed_emoji":"🧬","tokens_out":6301,"duration_ms":54963,"temperature":0.7,"pith_summary":"The paper sets out to replace manual trial-and-error in artificial-life research with automated search driven by foundation models. Its central claim is that a vision-language model that embeds rendered simulation frames can judge which simulations look like cells, organisms, or ecosystems, stay persistently interesting over time, or fill a diverse atlas of behaviors. If true, this would let researchers specify what they want to see, such as 'a self-replicating pattern', and let the search find the simulation rules that produce it, across substrates as different as Boids, Lenia, and cellular automata. The payoff would be a general, substrate-agnostic pipeline for discovering lifelike worlds and for quantifying qualities like open-endedness that have resisted formal definition.","feed_headline":"AI search finds new lifeforms in five simulation worlds","feed_subtitle":"ASAL automates the hunt for lifelike simulations, finding open-ended cellular automata and new flocking patterns without hand-crafted rules.","key_machinery":"The load-bearing object is the vision-language embedding of a rendered simulation frame. CLIP, and in the ablation DINOv2, maps each frame to a representation space where text prompts can be compared with images; ASAL then defines three objectives in that space: text-image alignment for targets, historical nearest-neighbor similarity for open-endedness, and nearest-neighbor distance for illumination. A renderer is part of the substrate parameterization for cases like Neural Cellular Automata where the raw state is not visually interpretable. The search optimizers are Sep-CMA-ES for single targets, Adam with backpropagation through time for temporal NCA targets, brute force for the 262,144 life-like cellular automata, and a custom genetic algorithm for illumination.","core_discovery":"ASAL is a three-mode search procedure. In supervised-target mode it maximizes the CLIP similarity between a rendered simulation frame and a text prompt, optionally at multiple timesteps to produce temporal sequences. In open-endedness mode it minimizes the similarity between each frame and its closest historical frame in CLIP space, treating sustained historical novelty as the signature of open-endedness. In illumination mode it searches for a population of simulations whose frames are far from their nearest neighbor in CLIP space. The paper reports that this procedure rediscovers known lifeforms, finds previously unseen Lenia and Boids organisms, and identifies life-like cellular automata whose CLIP-space trajectories keep diverging, which the authors describe as open-ended in the same sense as Conway's Game of Life. It also uses the same embeddings to quantify nonlinearity in parameter interpolation, the 'more is different' particle-number threshold in Particle Life, parameter sensitivity, and the time at which a Lenia simulation plateaus.","pith_inferences":["A natural next test is whether the open-endedness score tracks human judgments: if human raters are shown pairs of cellular automata and asked which keeps getting more interesting, their choices should match the CLIP-space novelty ranking for the central claim to carry over to perceived open-endedness.","The same pipeline could be applied outside ALife, for example to search hypergraph update rules or physical model parameters for behavior that an embedding model finds natural, since the only requirement is a renderable state and an embedding space.","Temporal prompts may be a route to directed counterfactual evolution: specifying a sequence like 'primordial soup', then 'predator-prey', then 'alien animals' tests whether a substrate can host a chosen evolutionary trajectory, not just a single final state.","If future video-language models replace CLIP, the open-endedness and illumination objectives could be computed on whole trajectories rather than single final frames, which would likely sharpen the distinction between genuinely open-ended and merely varied simulations."],"forward_implications":["Researchers can request a phenomenon in plain language, such as 'a caterpillar' or 'a network of neurons', and recover a simulation that produces it, as shown in Lenia, Boids, and Particle Life.","Searching for historical novelty in CLIP space can serve as a concrete, computable stand-in for open-endedness on any substrate that renders to images.","Illumination produces a 'simulation atlas' that maps the space of possible behaviors of a substrate, organized by visual similarity rather than by hand-picked descriptors.","The same embedding-based similarity can become a quantitative instrument: measuring parameter nonlinearity, ranking parameter importance, and detecting when a simulation has reached a static plateau.","Because the method is agnostic to the choice of foundation model and substrate, the same search recipes can be re-run as better vision models and new substrates appear."],"supporting_citations":[{"why":"Supplies CLIP, the vision-language embedding model that scores target prompts and provides the representation space for novelty and diversity.","marker":"[Radford et al., 2021]"},{"why":"Defines the Lenia continuous cellular automaton substrate where ASAL discovers previously unseen lifeforms.","marker":"[Chan, 2018]"},{"why":"Defines the Boids flocking model used as another substrate for discovering new collective behaviors.","marker":"[Reynolds, 1987]"},{"why":"Provides the Neural Cellular Automata substrate and the backpropagation-through-time training procedure used for temporal target search.","marker":"[Mordvintsev et al., 2020]"},{"why":"Supplies the LeniaBreeder codebase that defines the Lenia search space and the quality-diversity baseline ASAL extends.","marker":"[Faldor and Cully, 2024]"},{"why":"Defines the life-like cellular automata rule space of 262,144 simulations that ASAL brute-force searches for open-endedness.","marker":"[Wójtowicz, 2001]"},{"why":"Supplies the novelty-search principle that the open-endedness objective adapts: historical novelty in a chosen representation space.","marker":"[Lehman and Stanley, 2011a]"},{"why":"Provides the UMAP projection used to visualize and organize the discovered simulation atlas.","marker":"[McInnes et al., 2018]"}],"fun_headline_variants":["AI automates artificial life search, finds new organisms","Foundation models reveal new lifeforms in simulations","AI finds open-ended lifeforms in five simulation worlds","Automated search uncovers unprecedented life-like simulations","AI discovers new artificial lifeforms across five worlds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the internal representation a vision-language model builds of a rendered simulation frame matches what a human would find interesting, novel, or diverse, so that maximizing novelty or diversity in that representation yields genuinely lifelike discoveries instead of patterns that merely fool the model.","fun_headline_variants_meta":{"raw":{"variants":["AI automates artificial life search, finds new organisms","Foundation models reveal new lifeforms in simulations","AI finds open-ended lifeforms in five simulation worlds","Automated search uncovers unprecedented life-like simulations","AI discovers new artificial lifeforms across five worlds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3457,"prompt_tokens":977,"completion_tokens":2480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2418}},"tokens_in":593,"tokens_out":2480,"duration_ms":14386,"temperature":1.0,"reasoning_tokens":2418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:07:51.804756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blind human-preference study would settle the load-bearing assumption. Take the highest-scoring open-ended cellular automata from the brute-force search and an equal number of randomly chosen automata, render their rollouts, and ask human raters which are more interesting or more lifelike. If preference does not track the ASAL open-endedness score, or if humans find the random automata just as engaging, then the claimed human alignment of the foundation-model representation is not doing the work.","supporting_citations":[{"cited_title":"bird-like objects","cited_arxiv_id":null,"evidence_quote":"Defines the Boids flocking model used as another substrate for discovering new collective behaviors."}],"review_version":1}