{"id":"c22e9f9c-75ca-4b97-a739-baf32468e8c9","arxiv_id":"2608.10323","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":13,"one_line_summary":"In a spatial artificial life simulator, rankings between evolution-based and RL-based training regimes depend on architecture and evaluation context, with no universal winner.","lead":"The paper builds a simulated world where neural-network cells evolve, learn, and compete on a grid, then freezes saved champions and pits them against each other under different ecological rules. It finds that the apparent best training regime flips depending on the network architecture and the matchup, so no single regime is best across settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frozen evaluation substitutes standardized context/movement heads for trained per-cell heads; since EvoEvo/EvoRL can evolve these heads while RLRL cannot (Sec. 3.6), the architecture-conditioned pairwise ordering may be a reconstruction artifact rather than a regime property.","rationale":"The reader's weakest_assumption targets exactly this step: the saved artifact plus standardized reconstruction must fairly represent each run's ecological competence. I agree, and I sharpen it: the reconstruction is not merely a missing-state approximation, because the missing heads have regime-dependent training histories. EvoEvo and EvoRL can mutate context/movement heads; RLRL cannot. So a single reconstruction policy is a differential intervention that could plausibly manufacture or invert architecture-conditioned patterns. The paper does important service by reporting the reconstruction, the 2:1 context weighting, the n=3 independent level, the survival floor, and known defects; those disclosures are why this is not a rejection. But the positive empirical claim rests on an untested neutrality assumption. The proposed sensitivity check is cheap because the frozen runner is public. If the patterns survive under alternative reconstructions, the central claim is much stronger; if they do not, the paper should present the finding as a property of the standardized evaluation protocol, not of update-and-inheritance regimes. Thus I keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":16678,"tokens_out":10244,"duration_ms":105764,"concrete_test":"Using the existing 18 artifacts and the public frozen-evaluation runner, rerun the 135 pairwise jobs under three standardized reconstruction variants for the context-strategy and movement heads: (i) zero-output heads; (ii) fixed random heads sharing one seed; (iii) heads drawn from the training-time initialization distribution. As a separate check, retrain at least three runs per condition while saving full context/movement heads and compare frozen outcomes using true trained heads versus the standardized reconstruction. Record whether the three directional patterns from Table 6 survive in at least two of the three reconstruction variants and in the full-state comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical pattern is the architecture-conditioned pairwise ordering (MLP4 beats Wide128 within each regime; EvoEvo>EvoRL>RLRL within MLP4; RLRL>EvoRL>EvoEvo within Wide128), read off frozen pairwise AUCs of saved artifacts. Section 3.6 states that each legacy artifact stores only the elite cell's action-network W1/W2 matrices, not the per-cell context-strategy head or Boid movement head; formal evaluation reconstructs these omitted fields from the seed-defined context under one standardized policy applied to every condition. Those heads are not neutral accessories: the same section says they mutate only when global evolution is enabled, so EvoEvo and EvoRL can adapt them across generations while RLRL cannot. Replacing evolved heads with a common reconstruction is therefore an unequal intervention: evolutionary regimes lose any co-adapted head, RLRL loses heads it never tuned. The paper notes 'No context-head or movement ablation was performed' (Sec. 3.6) and lists reconstruction in Sec. 5.5, but never checks whether the standardized policy is neutral across regimes and architectures. Without that check, the majority orderings could be generated by the reconstruction policy interacting with action-network weights rather than by the update-and-inheritance regime itself. The negative claims (no universal winner, variation across artifacts/contexts) are cushioned by raw variability, but the positive architecture-conditioned ranking is the load-bearing unvalidated step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Neuroevolution Arena, a GPU-accelerated spatial artificial-life system in which each cell carries an independent neural network, and compares three update-and-inheritance regimes (EvoEvo, EvoRL, RLRL) crossed with two architectures (MLP4 and Wide128). Each condition is trained for 50,000 generations in three runs, and one elite artifact per run is saved. A formal frozen-evaluation protocol then evaluates these artifacts in 198 jobs: 135 aligned-run pairwise competitions, 9 six-way melees, and 54 survival runs, across three seed-defined contexts. The authors report higher recorded training fitness for RL-enabled regimes, architecture-conditioned pairwise majority patterns (EvoEvo > EvoRL > RLRL within MLP4; RLRL > EvoRL > EvoEvo within Wide128; MLP4 > Wide128 in all three regimes), substantial artifact dependence, a complete floor in the prespecified survival endpoint, and no universal winner. They contribute an audit-tracked nested protocol that separates training-run artifacts from evaluation contexts and explicitly avoids claims of a universal ranking or a demonstrated Baldwin Effect.","tokens_in":16994,"tokens_out":10666,"duration_ms":103757,"significance":"The paper is unusually transparent about its limitations: it reports n=3 independent runs, 4-of-15 unanimous condition pairs, 17-of-45 context-consistent blocks, a complete survival floor, and known implementation defects (warmup reward leak, stale mate-selection state), and it ships code and data. The methodological contribution (aligned-run pairwise design, explicit separation of computational jobs from statistical replicates, prespecified floor endpoint) is valuable to the artificial-life and neuroevolution communities. If the main empirical pattern were robust, the demonstration that ecological rankings are architecture- and artifact-dependent would be a useful cautionary result. However, the central architecture-conditioned ordering currently rests on an unvalidated reconstruction of un-saved context and movement heads, which weakens the inference from training regime to ecological outcome.","major_comments":[{"comment":"The central pairwise orderings are not yet attributable to the update-and-inheritance regimes. The saved artifacts contain only W1/W2 action-network weights; formal evaluation reconstructs the context strategy head and movement head under \"one standardized policy applied to every condition\" (Sec. 3.6). Immediately before that, the paper states these heads \"mutate only when global evolution is enabled,\" so EvoEvo and EvoRL had the opportunity to co-adapt them during 50K generations while RLRL did not. Replacing evolved heads with a common reconstruction is therefore an unequal intervention: EvoEvo/EvoRL lose any co-adapted heads, RLRL loses heads it never tuned. The paper notes that no context-head or movement ablation was performed and lists the reconstruction in Sec. 5.5, but it still presents the 2-1 majority orderings of Sec. 4.3 as the main ecological pattern. Without a sensitivity analysis over reconstruction policies (e.g., zero context input, random heads, or training the heads under each regime), the architecture-conditioned ranking could be an artifact of the standardized policy interacting with the saved action-network weights.","section":"Sec. 3.6 and Sec. 4.3"},{"comment":"The aligned-run design leaves the condition ordering at the mercy of three arbitrary run pairings, and with n=3 a single strong artifact can change a 2-1 majority. The paper correctly reports that only 4 of 15 condition pairs are directionally unanimous and 17 of 45 pair-by-run blocks are context-consistent, but those figures are relegated to descriptive caveats rather than used to bound the main claim. To make the architecture-conditioned pattern load-bearing, the authors should report the full cross-run artifact matrix (or at least a leave-one-run-out analysis of Table 6) and show that the within-architecture regime orderings and the MLP4-beats-Wide128 majority survive all three alignments. Absent that, the claim \"the pairwise majority direction of the three regimes changes with architecture\" describes three specific matched pairs, not the conditions.","section":"Sec. 4.3 and Table 6"}],"minor_comments":[{"comment":"The line \"Preprint. This manuscript has not undergone journal peer review.\" should be removed if this is submitted to a journal.","section":"Title page"},{"comment":"The sentence \"Formal evaluation reconstructs omitted ecological state under three seed-defined contexts\" is inaccurate; the evaluation reconstructs omitted controller fields (context and movement heads), whereas the ecological state is newly initialized from the seed context.","section":"Sec. 5.5"},{"comment":"The phrase \"mean AUC log-ratio over the three nested contexts\" could be misread as the mean of log-ratios; clarify that it is the mean of the three context-level log-ratios within each aligned run block.","section":"Table 6 caption"},{"comment":"The claim that the update-regime contrast is \"about 11 times larger\" than the architecture contrast would be easier to verify if the actual marginal means and standard errors were reported in the text rather than only the approximate magnitudes.","section":"Sec. 5.2"},{"comment":"The Hugging Face dataset and model links would benefit from versioned persistent identifiers, as recommended at the end of the manuscript.","section":"Artifact availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical pattern needs one additional validation step (reconstruction-policy sensitivity) before the architecture-conditioned ordering can be taken as a property of the training regimes. If the authors provide that analysis, the paper could become acceptable; the methodological protocol itself is a real contribution. I would not reject on the basis of the acknowledged n=3 limitation, but the reconstruction issue is load-bearing because the saved artifacts are incomplete and the shared reconstruction policy is not shown to be neutral."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Neuroevolution Arena preprint. The short version: it's a serious, well-bounded empirical study with a genuinely useful evaluation protocol, but the headline architecture-conditioned ranking may be an artifact of how the authors reconstruct missing controller components during frozen evaluation. Don't take the ranking at face value; the protocol itself is worth engaging with.\n\nWhat's new: per-cell independent neural networks in an artificial life setting, contrasted with the usual shared-weight NCA. The 3x2 factorial over update regimes (EvoEvo, EvoRL, RLRL) and architectures (MLP4, Wide128) is clean. The frozen evaluation design—aligned run blocks, contexts nested within blocks, 198 jobs with n=3 independent runs—is a good template. The paper reports a null survival endpoint, discloses known implementation defects (warmup reward leak, stale mate selection), and refuses to draw Baldwin-effect conclusions. That's honest, and the code/data links look real.\n\nThe soft spot is the standardized reconstruction. The saved artifacts only contain W1/W2 action-network weights; the per-cell context-strategy and movement heads are reconstructed under one policy for all conditions. Those heads are not neutral: they evolved in EvoEvo and EvoRL but not in RLRL, and no ablation was done. So the architecture-conditioned ordering (MLP4 beats Wide128; EvoEvo>EvoRL>RLRL in MLP4 but reversed in Wide128) could reflect the reconstruction policy interacting with action weights, rather than the regimes themselves. The authors note this in Sections 3.6 and 5.5, but they don't test it. That's the load-bearing missing check.\n\nOther weaknesses are more minor: the 2:1 cooperation/attack context mixture is disclosed but makes the pooled result protocol-specific, and n=3 runs per condition is thin. But these are acknowledged.\n\nWho should read this: anyone building artificial life evaluations or comparing neuroevolution with RL. The protocol and the honesty to the null result are valuable even if the empirical reversal fades with proper artifact saving.\n\nMy recommendation: send it to peer review. A referee should ask for either a reconstruction-ablation analysis or saved complete artifacts, and possibly a full cross-run design. That's revision-level work, not a desk reject. I'd cite the protocol once the reconstruction question is resolved.","headline":"Honest evaluation-protocol paper; the headline ranking may be a reconstruction artifact, but the protocol merits peer review.","tokens_in":17548,"tokens_out":3991,"would_cite":true,"duration_ms":37794,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained controllers do not keep their rank when moved from training into a shared ecology.","keywords":["artificial life","neuroevolution","reinforcement learning","competitive ecology","neural architectures","evaluation methodology","neural cellular automata","Baldwin effect"],"falsifier":"Re-run the frozen evaluations with complete saved population snapshots, including context and movement heads, for all nine cross-run artifact pairings per condition pair, with cooperation and attack contexts balanced rather than 2:1; if a single regime wins consistently across both architectures and most artifacts, the paper’s central claim of architecture-conditioned, artifact-dependent outcomes would be refuted.","tokens_in":16430,"feed_emoji":"🧬","tokens_out":6291,"duration_ms":58452,"temperature":0.7,"pith_summary":"The paper asks how rankings among neural controllers change when they are scored by training fitness versus by survival and competition in a shared artificial world. It builds a GPU-accelerated grid ecology in which each cell carries its own independent neural network, and compares three update-and-inheritance regimes crossed with two architectures over 50,000 generations. Its central finding is that no regime wins everywhere: training fitness favors RL-enabled regimes, but pairwise ecological outcomes flip with architecture and vary across saved training-run artifacts and ecological contexts. A prespecified survival endpoint yields a complete floor, so the study reports a null result rather than promoting a winner. The contribution is a nested evaluation protocol that separates training-run artifacts from evaluation contexts and makes their separate sources of variation visible.","feed_headline":"Regime rankings flip when controllers enter a shared ecology","feed_subtitle":"A 198-job frozen-evaluation study finds winners depend on architecture, saved artifact, and context.","key_machinery":"The central object is Neuroevolution Arena, a GPU-accelerated 100×100 toroidal grid of up to 10,000 cells, each holding its own action network (a perception layer $W_1$ and a three-layer decision network $W_2$) plus per-cell context-strategy and movement heads. Regimes differ in which layers update by evolutionary crossover/mutation or by lifetime reinforcement learning: EvoEvo evolves both, EvoRL evolves $W_1$ and learns $W_2$, and RLRL learns both while retaining reproduction and selection. The argument is carried by an aligned-run frozen-evaluation design: one saved elite artifact from each of 18 training runs enters 198 computational jobs (135 pairwise, 9 six-way, 54 survival), with three seed-defined ecological contexts (two cooperation-permitting, one attack-permitting) nested inside three matched training-run blocks, and pairwise effects measured by $\\log((\\mathrm{AUC}_A+1)/(\\mathrm{AUC}_B+1))$. This design is what exposes artifact dependence and architecture-conditioned rankings instead of treating evaluation as a fixed test set.","core_discovery":"In the tested 3×2 design, RL-enabled regimes reach higher recorded training fitness (EvoRL 335K and RLRL 299K versus EvoEvo 150K for MLP4), but that advantage does not carry into ecological competition. Under the aligned-run frozen-evaluation protocol, EvoEvo×MLP4 wins 11 of 15 condition-by-aligned-run blocks and holds majorities against all five opponents, while EvoEvo×Wide128 wins only 2. The within-architecture majority order reverses: EvoEvo > EvoRL > RLRL on MLP4, but RLRL > EvoRL > EvoEvo on Wide128, and MLP4 beats Wide128 within every regime by 2-to-1 aligned-block majorities. Only 4 of 15 condition pairs keep the same direction across all three aligned run blocks, and none of the 54 survival jobs recovers to the prespecified 5,000-cell threshold after a 97% shock. The paper concludes that ecological outcomes in this system depend jointly on controller configuration, saved training-run artifact, opponent set, and evaluation context, and that it does not identify a universally superior regime, a survival leader, or a demonstrated Baldwin Effect.","pith_inferences":["If the pattern generalizes beyond this system, benchmark rankings of evolutionary and reinforcement-learned controllers may be poor predictors of multi-agent or ecological competence, and single-environment evaluations could systematically mislead.","The pooled effect weights cooperation-permitting contexts 2:1 over the attack-permitting context, so a balanced design reporting cooperation and attack outcomes separately could reveal sharper regime-by-context interactions that the current mixture conceals.","Because each artifact stores only one elite cell’s action-network weights and reconstructs context and movement heads under one standardized policy, evaluating complete population snapshots or ensembles of elites from each run might change pairwise outcomes even without changing training.","The absence of a demonstrated Baldwin Effect is specific to the implemented prescribed layer split; evolving the learning schedule or plasticity rate would be the natural next test, though it is beyond this paper’s data."],"forward_implications":["Higher recorded training fitness does not imply ecological robustness: RL-enabled regimes lead in training fitness but not in pairwise or survival outcomes.","Apparent winners in competitive artificial life are conditional: the same controller type can lead in one architecture and trail in another under the same protocol.","Evaluation designs that average over saved artifacts or ecological contexts can hide or even invert rankings, so matched-run and balanced-context designs are needed.","A non-discriminating survival threshold is informative as a null result: a 97% shock is too severe for recovery to 5,000 cells under these conditions, and future studies must calibrate thresholds on separate pilot data.","In this system, no single update-and-inheritance regime can be recommended as universally superior; the interaction with architecture and evaluation context is the finding."],"supporting_citations":[{"why":"Supplies the computational Baldwin-effect model that motivates the EvoRL layer split and against which the paper’s boundary of no demonstrated Baldwin Effect is drawn.","marker":"(Hinton and Nowlan, 1987)"},{"why":"Defines the shared-weight neural cellular automaton that Neuroevolution Arena departs from by giving every cell independent parameters.","marker":"(Mordvintsev et al., 2020)"},{"why":"Establishes neuroevolution as the lineage on which the EvoEvo and EvoRL regimes build their evolutionary update rules.","marker":"(Stanley and Miikkulainen, 2002)"},{"why":"Shows that supervised rankings shift with random seeds and early stopping, motivating the paper’s scrutiny of artifact and context dependence.","marker":"(Dodge et al., 2020)"},{"why":"Provides the competitive-evolution precedent for evaluating controllers through opponents in a shared world.","marker":"(Sims, 1994)"}],"fun_headline_variants":["Training champs lose in shared ecology","Ecology flips regime rankings","Winner depends on architecture and artifact","No universal winner in neuroevolution arena","Survival floor: none recover after shock"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume that each saved elite artifact—one cell’s action-network weights from a training run, with context-head and movement weights rebuilt under a single standardized policy—fairly represents that run’s ecological competence.","fun_headline_variants_meta":{"raw":{"variants":["Training champs lose in shared ecology","Ecology flips regime rankings","Winner depends on architecture and artifact","No universal winner in neuroevolution arena","Survival floor: none recover after shock"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2466,"prompt_tokens":1004,"completion_tokens":1462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":620,"tokens_out":1462,"duration_ms":9574,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:17.345218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the frozen evaluations with complete saved population snapshots, including context and movement heads, for all nine cross-run artifact pairings per condition pair, with cooperation and attack contexts balanced rather than 2:1; if a single regime wins consistently across both architectures and most artifacts, the paper’s central claim of architecture-conditioned, artifact-dependent outcomes would be refuted.","supporting_citations":[],"review_version":1}