{"id":"febb56f9-41d2-4873-95bf-755e395e296f","arxiv_id":"2505.05863","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Words as organisms in an AI-judged battle royale evolve toward semantically 'strong' animal names, showing diverse and sometimes punctuated dynamics.","lead":"This paper builds a simple artificial ecosystem where each agent is a word produced by an AI language model, and nearby words 'fight' based on the AI's opinion of which is stronger, with the loser replaced. Starting from familiar animal names, the population drifts toward words the AI considers powerful (blue whales, tigers, extinct giants), and the authors propose this as a way to study open-ended evolution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Word order in the competition prompt is not randomized; unmeasured LLM positional bias could make the cached payoff matrix an artifact of agent turn order, undermining the selection claim.","rationale":"The paper's central claim requires the LLM's pairwise judgments to constitute a stable and meaningful fitness function. The reader correctly identified this assumption as the weakest. I sharpen it: the protocol as written fixes word order (own word first), and the dictionary cache freezes the first-encounter result. Positional bias is a well-documented confound in LLM comparison prompts, so the outcome of a pair can depend on which agent happened to initiate the first encounter. This makes the payoff matrix partly a function of stochastic update order, not just of the words' meanings. This is load-bearing because the experiments' main qualitative findings (convergence to large/specialized species, punctuated equilibrium) are reported without controls; if a substantial part of the 'selection' is actually position-driven, the headline claim that LLMs can serve as both mutation source and selection arbiter is not supported by the reported results. The concern is concrete and testable with the released code and data, and it can be settled by an order-randomization control plus a simple flip-rate measurement. I credit the authors for a clear model description, released code/data, and an honest preliminary framing; the issue is a missing control rather than an internal inconsistency. The reader's CONDITIONAL verdict remains appropriate, conditional on this test being run and reported. If the flip rate is high or the randomization changes the qualitative outcomes, the paper would need major revision or rejection.","tokens_in":7357,"tokens_out":6074,"duration_ms":66308,"concrete_test":"Run the released code with the competition prompt modified to randomize the order of the two words (e.g., alphabetical or coin flip) while holding all other settings fixed; compare the resulting frequency trajectories and final dominant species in Figures 3-5 to the original. Additionally, for a fixed set of 200 word pairs that appear in the experiments, call the Gemma-2 competition prompt with both word orders under identical generation settings and compute the flip rate (fraction of pairs whose winner changes with order). If the dominant-species outcomes are qualitatively unchanged and the flip rate is near zero, the concern is refuted; if they diverge or the flip rate is non-negligible, the cached payoff matrix is contaminated by positional bias and the evolutionary trajectories may largely reflect agent processing order rather than word semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the LLM competition prompt introduces an unmeasured positional bias that may be doing the work attributed to ecological selection. In Section 2 and Figure 2c, the competition query is \"Which one is stronger \\\"#word1#\\\" or \\\"#word2#\\\"?\", and the text states that an individual \"compete[s] with their own word against the opponent's word\", so the initiating agent's word is always placed first. LLM pairwise comparisons are known to be sensitive to token order. Because the first time a given word pair is encountered, the initiator is the agent that happens to act first that step, the cached outcome for that pair depends on the stochastic agent-update order, not on a stable property of the words. The dictionary then freezes this order-dependent result for the rest of the run. If Gemma-2 favors the first-mentioned word even slightly, the payoff structure is partly determined by who initiates, and the \"selection\" driving convergence to large/extinct species in Figures 3-5 could be an artifact of turn order rather than word semantics. The paper does not report any test for positional bias, and the released materials do not appear to include an order-randomized control. This is not a claim that the authors acted improperly; it is a gap in the evidence that directly affects whether an LLM can \"act as both the source of novelty and the arbiter of selection.\"","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an agent-based model in which each agent carries a single word (e.g., an animal species name) as its genome, moves on a two-dimensional toroidal grid, and interacts with neighboring agents. The outcome of each pairwise interaction is determined by an LLM (Gemma-2) answering the prompt 'Which one is stronger \"#word1#\" or \"#word2#\"?' with the winner's word replacing the loser's; mutations are also generated by the LLM when asked to list similar but different species. Two sets of experiments are reported: ten 300-step trials with 80 agents, and one 2000-step run with 200 agents. The authors report qualitative observations of word replacement, punctuated equilibrium, and convergence to large, extinct, or poisonous species, and interpret this as evidence that LLMs can serve as both the source of novelty and the arbiter of selection in an evolutionary ecology model.","tokens_in":7663,"tokens_out":5862,"duration_ms":59685,"significance":"If the central claim is supported, the paper makes a novel contribution by using an LLM simultaneously as a mutation generator and a fitness oracle in an evolutionary game-theoretic setting, potentially expanding the scope of open-ended evolution research. The model is simple, clearly described, and the accompanying code and data release (Figshare) is a reproducible and transparent asset. However, the evidence is largely qualitative, the long-term result rests on a single run, and a likely positional bias in the competition prompt directly threatens the interpretation of the LLM as a semantic fitness function. With appropriate controls and quantitative analyses, the model could become a useful platform for studying LLM-driven evolutionary dynamics; as it stands, the paper is better viewed as a preliminary report requiring substantial additional evidence.","major_comments":[{"comment":"The competition prompt always places the initiating agent's word first: \"Which one is stronger \\\"#word1#\\\" or \\\"#word2#\\\"?\" and the outcome for a word pair is cached after the first encounter. If Gemma-2 exhibits any positional bias in pairwise comparisons, the cached payoff matrix will depend on the stochastic agent-update order at the first meeting rather than on word semantics, and the claim that the LLM acts as an ecological selector would be compromised. The paper provides no order-randomized control or test for positional bias; I request an experiment with randomized word order and a report of the direction and magnitude of any bias.","section":"Section 2, Figure 2c"},{"comment":"The central claims of 'punctuated equilibrium' and 'emergence and coexistence of diverse species' are supported only by visual inspection of UMAP projections and frequency plots. No quantitative measures of diversity, turnover, or punctuation are provided, and the long-term experiment (Figure 5) consists of a single run. To justify the general conclusions, the authors should supply repeated long-term runs, diversity indices, and statistical tests, or explicitly reframe the results as qualitative observations from a preliminary model.","section":"Section 3, Figures 3-5"},{"comment":"No control conditions are reported. Without a baseline using a random payoff matrix or a fixed non-LLM ranking, it is not possible to distinguish the effect of the LLM's semantic knowledge from the dynamics of the spatial replacement process itself (e.g., the tendency for local populations to become homogeneous or for transient species to be replaced). Adding such controls is essential for the claim that the observed dominance of large or extreme species arises from the LLM's evaluation of word meanings.","section":"Section 3 (entire experimental section)"}],"minor_comments":[{"comment":"The phrase 'infinite options' overstates the model's capability; since the word space is generated by a finite LLM vocabulary and list sizes, 'unbounded' or 'open-ended' would be more accurate.","section":"Abstract and Introduction"},{"comment":"The figure is dense and difficult to read at the available resolution; the traces of average word vectors are hard to distinguish from the background gray words. Consider separate panels per trial or clearer trajectory markers.","section":"Figure 3"},{"comment":"The figure caption refers to 'word frequency in the final step' while the text refers to 'top 10 frequent species in the final step'; please align the terminology and state clearly what the frequency plot shows.","section":"Figure 4 and accompanying text"},{"comment":"The sentence 'As for the top 10 species, With the exception of the Siberian tiger...' has a grammatical error (capitalization after a comma) and becomes a run-on; it should be split into cleaner sentences.","section":"Section 3, paragraph after Figure 4"},{"comment":"The paper does not specify the LLM's decoding parameters (e.g., temperature, top-p, random seed), even though these affect reproducibility of the LLM-generated judgments and mutations. Please add these details or point to them in the code repository.","section":"Section 2, Model"},{"comment":"The text states 'whale obtained X and eagle obtained Y, and whale won the game because X > Y', but X and Y are not defined; the reader is left to guess that they represent payoffs. Please clarify or omit the X/Y notation.","section":"Section 2, Figure 1d"}],"recommendation":"major_revision","confidential_remarks":"The positional-bias concern in the competition prompt is the primary risk to the paper's central claim; if a rerun with randomized word order changes the outcomes, the conclusions may need substantial revision. The paper is otherwise a reasonable preliminary report, but it currently lacks the controls and quantitative rigor expected for a journal publication. The code/data release is commendable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a plausible minimal model for using an LLM as both mutation source and fitness function in a spatial evolutionary game. The authors set up words as organisms on a grid, compete them pairwise via Gemma-2, and let winners overwrite losers. The novel bit is the combination — LLM-generated mutations plus LLM-judged competition inside an explicit spatial ecology. That combination isn't in the cited literature, and the paper shows it can produce rich, qualitative dynamics: 10 trials each yielding over a hundred unique species, and a long run with 3,704 distinct names, including parasites and fungi. The code and data are on figshare, which is real evidence and helps.\n\nWhat the paper does well: it's clearly written, the model is minimal and reproducible, and the authors are upfront that these are preliminary experiments. They also identify plausible next steps (draws, crossovers, environmental descriptions).\n\nThe soft spots are substantial, though. The central claims — punctuated equilibrium, convergence to large/extinct species, open-endedness — rest entirely on visual inspection of UMAP plots. There are no repeated runs with identical initial conditions, no measures of diversity or takeover times, and no baseline comparisons (e.g., random judgment, fixed payoff matrices, or word-order controls). The stress-test note is right: in the competition prompt, the acting agent's word is always placed first. Since agent order is random, the first encounter of any word pair caches a result that depends on who happened to act first that step. If Gemma-2 has any positional preference, the cached payoff matrix is partially arbitrary, and the \"selection\" driving the dynamics could reflect stochastic turn order more than word semantics. The same issue applies to any LLM pairwise comparison, and the authors don't report any control for it. This is a gap in the evidence, not an accusation.\n\nThere's also an inherent circularity: the LLM supplies the mutations and the fitness signal, so the dynamics are largely a projection of its priors. That's not fatal — the authors explicitly frame the goal as \"virtual\" evolution using LLM judgments — but it does mean the \"prediction\" that large/extinct species dominate is essentially a restatement of what Gemma-2 already believes about strength. The paper would be much stronger with a sensitivity analysis on the LLM prompt, temperature, and order.\n\nWho should read it: anyone working on LLM-based artificial life or open-ended evolution. It's a good starting point for discussion, but it should not be taken as a validated result. I'd send it to peer review, but with a clear expectation of major revision: add controls, repeated runs, and quantitative metrics.\n\nVerdict: worth reading, worth citing as an early example, but only if the revision addresses the positional-bias control and baselines.","headline":"A promising but under-evidenced demonstration of LLM-driven word evolution; the lack of order-randomized controls makes the selection signal partly suspect, but the model is worth a serious round of revision.","tokens_in":8148,"tokens_out":2932,"would_cite":true,"duration_ms":29805,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91A22"],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM can act as both mutation source and fitness judge, making word evolution open-ended.","keywords":["large language models","agent-based models","evolutionary game theory","artificial life","open-ended evolution","word ecology","spatial evolution","novelty generation"],"falsifier":"Re-run the full simulation twice with the same initial population and parameters, once using the cached LLM judgments and once using a fixed random tournament matrix over the same word pairs; if the random-tournament version still produces hundreds of species and punctuated equilibria, the LLM's semantic judgments are not the cause of the claimed evolution. A cheaper check is to query the same pair ('whale' versus 'eagle') many times with nonzero temperature and count how often the winner flips.","tokens_in":7183,"feed_emoji":"🧬","tokens_out":7631,"duration_ms":76119,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can replace both the mutation operator and the fitness function in an evolutionary system, making the space of possible strategies effectively open-ended. The proposed model places agents on a two-dimensional grid, gives each agent a short word as its genome, and resolves encounters by asking the LLM which word is stronger; the loser's word is overwritten by the winner's, and occasional mutations are generated by prompting the LLM for similar-but-different words. In experiments where 'strong animal species' are supposed to survive, populations that start from familiar animals diverge into terrestrial, marine, and extinct megafauna, with over a hundred species appearing per trial and dominance shifting in bursts. A longer, larger run produces coexistence of poison frogs, fungal pathogens, and malaria parasites, which the authors present as evidence that LLM-based novelty plus LLM-based selection supports open-ended evolution.","feed_headline":"Words evolve like species under an LLM fitness judge","feed_subtitle":"From blue whales to malaria parasites, a language model supplies mutation and selection in an open-ended ecology.","key_machinery":"The load-bearing mechanism is the LLM-as-oracle: a cached prompt call, 'Which one is stronger ...?', supplies the selection pressure for every pair of words, and a second prompt call supplies mutations by listing similar-but-different words. Because these calls draw on the LLM's open vocabulary, the set of possible strategies is effectively unbounded and the payoff structure is never fixed in advance, which is what the authors claim distinguishes this model from standard matrix-based evolutionary game theory. The spatial grid and the rule that occupied cells block movement add population viscosity, letting new words spread locally before they take over globally.","core_discovery":"On the paper's own terms, the central discovery is that an LLM can serve simultaneously as the source of new variants and as the arbiter of which variants spread, and that this minimal setup is enough to reproduce qualitative signatures of biological evolution: gradual drift interrupted by takeover events, ecological specialization, and the appearance of organism types that were never in the initial pool. The payoff matrix is never written down; it exists implicitly in the LLM's pairwise comparisons, so the strategy space is not fixed in advance. Under the prompt 'strong animal species survive,' each trial converges on a different large, specialized winner, and the long experiment shows that species substantially outside the initial category—toxin-producing amphibians, fungi, a malaria parasite—can appear and persist.","pith_inferences":["The paper leaves untested whether the dynamics rely on the LLM's semantic content or simply on having a mostly consistent, non-transitive tournament. An editorial conjecture is that cyclic dominance (A beats B, B beats C, C beats A) in the cached judgments could itself sustain diversity, so the 'ecology' may be a projection of the oracle's opinion graph.","A natural extension would be to replace the LLM judge with a randomly generated pairwise matrix and see whether similar punctuated equilibria appear; that would separate the contribution of the model's rules from the contribution of its world knowledge.","The same mutation-plus-judgment loop could be applied to non-biological word spaces, such as evolving product names or slogans against an LLM-elicited preference, since the mechanism never depends on animal-specific semantics."],"forward_implications":["If the LLM's pairwise judgments are stable enough, evolutionary game theory can move beyond finite, pre-specified strategy sets: new actions can arise mid-run and the payoff matrix can be implicit rather than explicit.","The same experimental apparatus can be pointed at a different selection criterion simply by changing the prompt (for instance 'cute' or 'witty'), so the model offers a cheap way to build artificial-life worlds with arbitrary fitness semantics.","The LLM's latent knowledge—about size, toxicity, habitat, extinction—becomes an implicit fitness landscape, meaning observed 'ecological' outcomes reflect the model's world model rather than an external biological truth.","The long experiment's coexistence of diverse species suggests that LLM-generated novelty plus spatial structure can sustain diversity, a combination the authors propose as a step toward open-ended artificial evolution."],"supporting_citations":[{"why":"The open-source chat LLM that supplies every competition judgment and mutation candidate, so the entire model runs on this oracle.","marker":"[14]"},{"why":"Introduces LLM-based crossover via few-shot prompting, the technique the mutation step extends to generate similar-but-different words.","marker":"[7]"},{"why":"Shows prompts themselves can evolve as genes, supporting the idea that LLM-generated variation can serve as an evolutionary operator.","marker":"[8]"},{"why":"Demonstrates LLMs generating new solutions for quality-diversity algorithms, the diversity-production role the word-mutation prompt plays here.","marker":"[9]"},{"why":"Prior agent-based model in which LLM-generated personality descriptions evolve under selection, a direct precursor to using words as evolving genomes.","marker":"[12]"},{"why":"Shows LLM multi-agent populations produce open-ended creative novelty, the phenomenon this paper extends to word-level ecological competition.","marker":"[13]"}],"fun_headline_variants":["LLM judges word survival in open-ended evolution","Language model evolves words into new species","Words mutate and compete, judged by an LLM","LLM-driven word evolution mimics punctuated equilibrium"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole process depends on the LLM answering 'Which one is stronger?' consistently enough to act as a fitness function; if those pairwise judgments are noisy, context-dependent, or unstable across re-queries, the observed evolutionary trajectories are partly artifacts of the oracle rather than results of the model's ecological rules.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges word survival in open-ended evolution","Language model evolves words into new species","Words mutate and compete, judged by an LLM","LLM-driven word evolution mimics punctuated equilibrium"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1300,"prompt_tokens":884,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":500,"tokens_out":416,"duration_ms":4628,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:53:38.887623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full simulation twice with the same initial population and parameters, once using the cached LLM judgments and once using a fixed random tournament matrix over the same word pairs; if the random-tournament version still produces hundreds of species and punctuated equilibria, the LLM's semantic judgments are not the cause of the claimed evolution. A cheaper check is to query the same pair ('whale' versus 'eagle') many times with nonzero temperature and count how often the winner flips.","supporting_citations":[{"cited_title":"Large language models as in-context AI generators for quality-diversity,","cited_arxiv_id":null,"evidence_quote":"Demonstrates LLMs generating new solutions for quality-diversity algorithms, the diversity-production role the word-mutation prompt plays here."},{"cited_title":"An evolutionary model of personality traits related to cooperative behavior using a large language model,","cited_arxiv_id":null,"evidence_quote":"Prior agent-based model in which LLM-generated personality descriptions evolve under selection, a direct precursor to using words as evolving genomes."},{"cited_title":"Collective innovation in groups of large language models,","cited_arxiv_id":null,"evidence_quote":"Shows LLM multi-agent populations produce open-ended creative novelty, the phenomenon this paper extends to word-level ecological competition."}],"review_version":1}