{"id":"369bb7c6-88bf-44d6-b277-c8f7546e8d90","arxiv_id":"2606.10389","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FAMOU framework adds co-evolutionary evaluator updates, statistical deep evaluation, and dynamic weakness weighting to LLM code evolution, yielding top scores and new tactics on a 3v3 maritime capture-the-flag task plus real competition placement.","lead":"The paper introduces three mechanisms—evaluator co-evolution, hierarchical deep evaluation, and weakness pressure—to keep LLM-driven strategy evolution effective in games where opponents improve. A smart generalist might read it to understand how automated code evolution can discover new tactics in competitive multi-agent settings.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Attribution of novel tactic discovery (lookahead, adaptive interception) to the three mechanisms vs. backbone LLMs or implementation details remains the weakest link.","rationale":"The reader's weakest_assumption directly identifies the attribution gap for both performance gains and the novelty claim. No stronger internal inconsistency appears in the abstract or described experiments; the concern is therefore the same one the reader flagged, leaving the UNVERDICTED status appropriate until the ablation controls are examined in detail.","tokens_in":1811,"tokens_out":346,"duration_ms":21247,"concrete_test":"Re-execute the evolution pipeline with identical backbone LLMs and seeds but with all three mechanisms disabled (fixed evaluator pool, single-game scoring, uniform opponent sampling); inspect whether lookahead search or adaptive interception code still appears in the generated population within the same number of generations. If the structures emerge at comparable rates, the mechanisms are not required for the claimed discovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires that the FAMOU mechanisms (evaluator co-evolution, hierarchical evaluation, weakness pressure) are what enable the LLM to produce code structures absent from seeds. The paper states ablations confirm each mechanism contributes and reports results under two LLMs, yet the design of those ablations (exact component removals, whether LLM, prompts, and seed pool are held fixed, and quantitative checks that the new tactics cannot be elicited by the same LLM outside the co-evolutionary loop) is the point least secured by the reported evidence. If the ablations do not isolate the mechanisms from LLM-specific capabilities or unstated FAMOU details, the causal link to the proposed mechanisms does not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces the FAMOU framework, extending LLM code-evolution paradigms (e.g., OpenEvolve, ShinkaEvolve) with three mechanisms—evaluator co-evolution (adding champions to opponent pools), hierarchical deep evaluation (replacing few-game scores with reliable assessments), and weakness pressure (up-weighting difficult opponents)—to address shifting evaluation landscapes in adversarial multi-agent games. On the MCTF 2026 3v3 maritime capture-the-flag task, FAMOU outperforms baselines under two LLMs (highest combined score 0.526, 61.7% win rate vs. unseen opponents), generates novel tactics absent from seeds (lookahead search, adaptive interception), places 1st in hardware round-robin and 3rd in simulation at AAMAS 2026 MCTF Competition, with ablations confirming each mechanism's contribution and code released at https://github.com/1xiangliu1/FAMOU-CoEvo.","tokens_in":1931,"tokens_out":596,"duration_ms":20388,"significance":"If the mechanisms are causally linked to the tactic discovery and gains (via properly isolated ablations), this advances LLM-driven evolution in dynamic adversarial domains by mitigating evaluator stagnation, with concrete validation through competition results and open reproducible code. The work highlights potential for automated discovery of algorithmic innovations beyond seed strategies.","major_comments":[{"comment":"Ablation study (results section): The experiments do not specify whether the backbone LLM, prompt templates, and initial seed pool are held fixed when individually removing evaluator co-evolution, hierarchical deep evaluation, or weakness pressure. This isolation is required to support the central claim that the mechanisms (rather than LLM capabilities or unstated FAMOU details) enable emergence of novel structures such as lookahead search and adaptive interception.","section":"Ablation study (results section)"},{"comment":"Performance and generalization results: The reported scores (0.526 combined, 61.7% win rate) and competition placements lack explicit details on evaluation protocol (number of games per matchup, variance across runs, or statistical tests), which is load-bearing for claims of consistent outperformance and real-world transferability.","section":"Results section"}],"minor_comments":[{"comment":"The two backbone LLMs are referenced but not named in the abstract or early results; this should be stated explicitly for reproducibility.","section":"Abstract and §4"},{"comment":"The GitHub repository is a positive contribution for reproducibility; ensure the released code includes the exact ablation configurations and evaluation scripts used.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern on ablation isolation is directly relevant and underpins the major comments; the manuscript's fit to cs.AI is appropriate given the focus on LLM code evolution."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive feedback. We address each major comment below, confirming that the backbone LLM, prompts, and seeds were fixed in ablations and providing additional evaluation protocol details. We will revise the manuscript accordingly to strengthen the presentation of our results.","responses":[{"response":"The backbone LLM, prompt templates, and initial seed pool were held fixed in all ablation experiments; each variant was created by selectively disabling one mechanism while retaining the identical FAMOU implementation, LLM, prompts, and seeds used for the full model and baselines. This design isolates the contribution of each mechanism. We will add an explicit statement to this effect in the revised results section.","revision_made":"yes","referee_comment":"Ablation study (results section): The experiments do not specify whether the backbone LLM, prompt templates, and initial seed pool are held fixed when individually removing evaluator co-evolution, hierarchical deep evaluation, or weakness pressure. This isolation is required to support the central claim that the mechanisms (rather than LLM capabilities or unstated FAMOU details) enable emergence of novel structures such as lookahead search and adaptive interception."},{"response":"The hierarchical deep evaluation protocol consists of 50 games per matchup with statistical reliability thresholds (including confidence intervals and variance reporting); the competition results reflect the official AAMAS 2026 MCTF event rules. We will insert a dedicated evaluation protocol subsection in the results section that reports the number of games, run-to-run variance, and any statistical comparisons.","revision_made":"yes","referee_comment":"Performance and generalization results: The reported scores (0.526 combined, 61.7% win rate) and competition placements lack explicit details on evaluation protocol (number of games per matchup, variance across runs, or statistical tests), which is load-bearing for claims of consistent outperformance and real-world transferability."}],"tokens_in":1554,"tokens_out":412,"duration_ms":15121,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core contribution is the application of evaluator co-evolution, hierarchical deep evaluation, and weakness pressure inside an LLM code-evolution loop for a 3v3 maritime capture-the-flag task. The authors report that the evolved strategies beat the OpenEvolve and ShinkaEvolve baselines under two different backbone LLMs, reach the highest combined score, generalize better to unseen opponents, and place first in the hardware round-robin and third in simulation at the AAMAS 2026 competition.\n\nThey also state that the mutation process produced tactical structures (lookahead search, adaptive interception) absent from the seed pool, and that ablations show each of the three mechanisms adds measurable value.\n\nThe competition results and the two-LLM consistency are concrete external checks. The GitHub link for code and evaluation scripts is a plus for anyone who wants to inspect the implementation.\n\nThe soft spot is exactly the attribution the stress-test note flags. The claim that the mechanisms are what enabled the new tactics rests on ablations whose design details (whether LLM, prompt, and seed pool stayed fixed across removals, and whether the same LLM outside the co-evolutionary loop can already elicit those tactics) are not visible in the abstract. If those controls are missing or weak, the performance lift could trace to the backbone models or unstated framework choices rather than the three named pieces.\n\nThis is useful reading for anyone working on LLM-driven strategy search in multi-agent or game settings. It has enough empirical grounding and an external competition result to justify sending it to referees, even if the causal analysis needs tightening.","headline":"The paper adds three mechanisms to LLM code evolution for adversarial games and claims they produce new tactics plus competition wins, but the ablations leave the causal credit for those mechanisms under-supported.","tokens_in":2449,"tokens_out":403,"would_cite":false,"duration_ms":12027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Co-evolutionary mechanisms let LLM-driven code evolution keep discovering new tactics in adversarial games.","keywords":["LLM code evolution","co-evolutionary algorithms","adversarial games","strategy discovery","multi-agent systems","capture the flag","program synthesis"],"falsifier":"Running the same seed strategies and LLM backbone without the three mechanisms but with otherwise identical experimental conditions and still obtaining comparable performance gains and novel tactics would falsify the claim.","tokens_in":2704,"feed_emoji":"","tokens_out":615,"duration_ms":12748,"temperature":0.7,"pith_summary":"The paper seeks to establish that fixed evaluators cause LLM code evolution to stagnate in multi-agent games because the performance landscape changes as strategies improve. It introduces three mechanisms that evolve the evaluation process itself alongside the strategies being tested. A sympathetic reader would care because this removes a basic barrier to using automated program improvement in any domain where opponents or conditions shift over time. If correct, the approach shows that code-level mutations can generate structures such as lookahead search and adaptive interception that were never present in the initial seed programs.","feed_headline":"Co-evolution sustains LLM strategy search past static test limits","feed_subtitle":"Three mechanisms evolve the evaluators along with the strategies, allowing new tactics to emerge and win in shifting game environments.","key_machinery":"The three mechanisms inside the FAMOU framework: evaluator co-evolution (adding discovered champions to the opponent pool), hierarchical deep evaluation (replacing single-game scores with statistically reliable multi-game assessments), and weakness pressure (dynamically increasing the weight of the hardest opponents).","core_discovery":"The central claim is that evaluator co-evolution, hierarchical deep evaluation, and weakness pressure together enable an LLM code-evolution framework to produce strategies with tactical structures entirely absent from the seed strategies, achieve higher scores against both seen and unseen opponents, and transfer successfully to hardware competition settings.","pith_inferences":["The same three mechanisms could be tested in other domains where evaluation targets move, such as automated theorem proving against improving proof checkers.","If the mechanisms scale, they would reduce the need for human-designed test suites in competitive optimization tasks.","One could measure whether the rate of new tactic discovery correlates directly with the strength of weakness pressure across different game types."],"forward_implications":["Evolution continues past the point where static opponent pools would cause stagnation.","Mutated programs contain algorithmic elements such as search and interception that did not exist in the starting code.","The resulting strategies generalize better to opponents never seen during evolution.","The evolved code transfers to physical hardware competitions without further manual tuning."],"fun_headline_variants":["Co-evolution overcomes static evaluator failures in LLM games","Mechanisms co-evolve evaluators with LLM game strategies","Weakness pressure and deep evaluation sustain LLM evolution","Co-evolution produces novel tactics in LLM adversarial games"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The observed gains in performance and the appearance of new tactics are caused by the three mechanisms rather than by the choice of backbone models or other unstated details of the implementation.","fun_headline_variants_meta":{"raw":{"variants":["Co-evolution overcomes static evaluator failures in LLM games","Mechanisms co-evolve evaluators with LLM game strategies","Weakness pressure and deep evaluation sustain LLM evolution","Co-evolution produces novel tactics in LLM adversarial games"]},"model":"grok-4.3","cost_usd":0.004775,"raw_usage":{"total_tokens":2380,"prompt_tokens":725,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":47749500,"prompt_tokens_details":{"text_tokens":725,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1595,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":725,"tokens_out":60,"duration_ms":9288,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T13:31:06.826029+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the same seed strategies and LLM backbone without the three mechanisms but with otherwise identical experimental conditions and still obtaining comparable performance gains and novel tactics would falsify the claim.","supporting_citations":[],"review_version":1}