pith. sign in

arxiv: 2604.14969 · v1 · submitted 2026-04-16 · 💻 cs.AI

Discovering Novel LLM Experts via Task-Capability Coevolution

Andrew Dai , Boris Meinardus , Ciaran Regan , Yingtao Tian , Yujin Tang This is my paper

Pith reviewed 2026-05-10 10:57 UTC · model grok-4.3

classification 💻 cs.AI
keywords LLM discoverycoevolutionmodel mergingsynthetic task generationcapability coverageopen-ended evolutionmodel populations
0
0 comments X

The pith

Coevolution of models and tasks produces archives of smaller LLMs whose combined expertise exceeds that of larger single models on benchmarks without any direct optimization.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that manually starting new training runs with fixed datasets limits the discovery of diverse capabilities. Instead, it lets LLMs and natural language tasks evolve together in one continuous process: models are merged to create variants, and new tasks are generated synthetically to challenge them. The resulting population of models collectively covers more skills across downstream benchmarks than curated collections or baselines, even though no benchmark data is used during evolution. Coverage keeps increasing over time as the archive grows and new tasks and models are added.

Core claim

AC/DC extends coevolution to LLM discovery by evolving both models via merging and tasks via synthetic data generation. This produces growing archives of LLMs that surpass larger models in capability while using less GPU memory. The populations achieve broader coverage of expertise than other curated models or baselines on downstream benchmarks without any explicit benchmark optimization. AC/DC improves coverage over time, continually innovates on tasks and models, and boosts performance in multi-agent best-of-N selection.

What carries the argument

The AC/DC loop that alternates model merging to create new LLM variants with synthetic task generation to produce fresh natural language challenges, thereby building an expanding archive of diverse capabilities.

If this is right

  • Collections of merged smaller models can exceed single larger models in total skill coverage while using less memory.
  • Capability diversity can increase indefinitely in one run without external benchmarks or human-curated data.
  • Continual addition of new models and tasks leads to ongoing innovation in both domains.
  • The archive improves results when used for multi-agent selection strategies such as best-of-N.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could reduce dependence on static pre-training datasets by letting capabilities emerge from internal coevolution.
  • If the generated tasks prove transferable, the approach might discover useful skills for domains lacking existing benchmarks.
  • The same coevolution structure could be applied to other model families or modalities to test whether coverage gains generalize.

Load-bearing premise

Synthetic task generation and model merging create genuinely novel and transferable capabilities rather than internal artifacts that fail to generalize outside the evolution process.

What would settle it

A test showing that the evolved model archive loses its coverage advantage when evaluated on a fresh set of real-world benchmarks whose task distribution was never used to generate synthetic data.

Figures

Figures reproduced from arXiv: 2604.14969 by Andrew Dai, Boris Meinardus, Ciaran Regan, Yingtao Tian, Yujin Tang.

Figure 1
Figure 1. Figure 1: Method Overview. AC/DC coevolves an increasing set of diverse LLMs alongside an increasingly diverse and complex set of tasks, measuring the discovered models’ capabilities. Our discovered collective of models (across different model families tested) covers more skills than baselines across a wide range of benchmarks. Moreover, AC/DC discovers improved single model performance (as seen by MMLU (Hendrycks e… view at source ↗
Figure 2
Figure 2. Figure 2: Algorithm Overview. AC/DC continuously coevolves a model (LLM) archive and a synthetic task archive. LLMs are evolved using model merging crossover, and weight noising as a mutation operation. Tasks are evolved using a large scientist LLM that transforms existing task descriptions to generate increasingly novel and complex tasks. Models are evaluated on this data. We then compute a skill vector (i.e., sign… view at source ↗
Figure 3
Figure 3. Figure 3: illustrates how our eight discovered models develop distinct performance profiles, with each model excelling in specific categories while performing differently across others, enabling them to function as complementary components of a collective intelligence. This specialization creates valu￾able Coverage patterns where models contribute unique capabilities to the ensemble. For instance, Model 4 may not ac… view at source ↗
Figure 4
Figure 4. Figure 4: Merged models unlock new capabilities. Higher Coverage means that our models solve tasks that baselines didn’t. These examples show a sample from MMLU, GSM8K, and GPQA, respectively, where none of the baseline models (math expert, code expert, reprompting the instruct model 8x, and the 72B model) solved the task, whereas at least one of our models did. 5.2 RESPONSE EXAMPLES AND DIVERSITY FROM MERGED MODELS… view at source ↗
Figure 5
Figure 5. Figure 5: Models in our Task Force give diverse answers. Two examples of synthetic tasks generated by AC/DC and the answers of 3 models in our Task Force. In the left example, we can see how all three models give different analogies. Moreover, Model 1 structures the analogy in a Python function. For the right example, we can see that our models provide 3 different implementations of the same optimal algorithm. 8 [P… view at source ↗
Figure 6
Figure 6. Figure 6: Adaptation types and Vendi score over time. For this experiment, we only enabled adaptations types to be making a task more difficult or completely novel. Moreover, we show the global Vendi Score (Vendi score of the global task archive) over time demosntreating increasing diversity in our task archive. D.3 TASK ARCHIVE NOVELTY OVER TIME [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: New models added to archive per generation. [PITH_FULL_IMAGE:figures/full_fig_p037_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Analysis of global task archive embedding space generated by AC/DC with Qwen 2. We represent each task by structuring its metadata using the template in Sec. F.3 and then embedding it using an embedding model (see Tab. 5). We then reduce the dimensionality of the embeddings using t-SNE (van der Maaten & Hinton, 2008). The clusters are automatically generated using HDBSCAN (McInnes et al., 2017) [PITH_FULL… view at source ↗
Figure 9
Figure 9. Figure 9: Evolution tree of AC/DC evolving the Qwen2-based seed model. Highlighted models are [PITH_FULL_IMAGE:figures/full_fig_p044_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Lineages of AC/DC evolved Qwen2-based models. All presented lineages are of models [PITH_FULL_IMAGE:figures/full_fig_p045_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of three seed models to the three fittest merged models on the global syn [PITH_FULL_IMAGE:figures/full_fig_p073_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Confusion matrix of synthetic tasks where all models merged and seed models failed and [PITH_FULL_IMAGE:figures/full_fig_p074_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qwen2.5 performance on 31 human-labeled OOD and synthetic tasks (see Sec. H), [PITH_FULL_IMAGE:figures/full_fig_p075_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Gibberish models detected via our gibberish filter for experiments with (a) Llama3 8B [PITH_FULL_IMAGE:figures/full_fig_p082_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Scaling trend with the number of models on our Qwen2.5 based experiment. [PITH_FULL_IMAGE:figures/full_fig_p089_15.png] view at source ↗
read the original abstract

Frontier model developers aim to train models continually to possess emergent, diverse capabilities. To extend capabilities, the current pre-training and post-training paradigm requires manually starting training runs with static datasets or reward functions every time. Addressing this limitation, our work pursues the insight that open-endedness (via the coevolution of models and tasks) can discover models with increasingly novel skills in a single run. We introduce a new model development framework that extends coevolution to large language model (LLM) discovery, open-ended \textit{Assessment Coevolving with Diverse Capabilities} (AC/DC). AC/DC evolves both LLMs via model merging and natural language tasks via synthetic data generation. AC/DC discovers growing archives of LLMs that surpass the capabilities of larger LLMs while taking up less GPU memory. In particular, our LLM populations achieve a broader Coverage of expertise than other curated models or baselines on downstream benchmarks, without \textit{any} explicit benchmark optimization. Furthermore, AC/DC improves Coverage over time, continually innovates on tasks and models, and improves performance in multi-agent best-of-N selection. Our findings highlight the potential of coevolution as a means of discovering broader sets of capabilities from base LLMs. Overall, AC/DC brings us one step closer to a profoundly new paradigm of LLM development, where continual improvements to the diversity of model capabilities can be accelerated by leveraging existing models as stepping stones to increasingly powerful models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces AC/DC (Assessment Coevolving with Diverse Capabilities), a coevolution framework that jointly evolves populations of LLMs via model merging and natural language tasks via synthetic data generation. It claims that the resulting LLM archives achieve broader expertise Coverage on downstream benchmarks than curated models or baselines, without any explicit benchmark optimization; that Coverage improves over time; that the process continually innovates on tasks and models; and that it yields gains in multi-agent best-of-N selection.

Significance. If the central empirical claims hold after rigorous validation, the work would be significant for shifting LLM development toward open-ended, automated discovery of diverse capabilities from base models, reducing reliance on static datasets and manual reward design. The coevolution approach and the reported memory-efficient expert populations are conceptually promising directions.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Experiments): The central claim that LLM populations achieve broader Coverage than baselines 'without any explicit benchmark optimization' is load-bearing, yet the manuscript provides no definition of the Coverage metric, no list of the specific downstream benchmarks, no statistical details (sample sizes, variance, controls for multiple testing), and no ablation that isolates the synthetic generator from benchmark-adjacent distributions. This makes it impossible to assess whether observed gains reflect genuine generalization.
  2. [§3.2] §3.2 (Task Generation): The synthetic task generator is described without diversity metrics, prompt templates, or controls (e.g., an ablation that replaces it with an independent task source or freezes it after initial generations). Without such tests, the reported Coverage improvements and continual innovation could arise from implicit distributional overlap or closed-loop leakage rather than novel, transferable capabilities.
  3. [§5] §5 (Results on multi-agent selection and Coverage trajectories): The improvements in best-of-N selection and the claim that Coverage increases over time lack baselines that hold the model-merging archive fixed while varying only the task generator (or vice versa). This confounds the contribution of coevolution and weakens the assertion that the framework 'continually innovates.'
minor comments (2)
  1. [§2] Notation for 'Coverage' is introduced in the abstract but never formally defined with an equation or pseudocode; a precise definition (e.g., fraction of tasks above a performance threshold across an archive) should be added in §2 or §3.
  2. [§1] The manuscript cites prior coevolution literature only sparsely; additional references to open-endedness work in evolutionary computation and LLM merging papers would clarify the novelty of extending these ideas to LLMs.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important areas for improving clarity and experimental rigor. We address each major comment point by point below and will revise the manuscript to incorporate the requested definitions, details, metrics, and controls.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): The central claim that LLM populations achieve broader Coverage than baselines 'without any explicit benchmark optimization' is load-bearing, yet the manuscript provides no definition of the Coverage metric, no list of the specific downstream benchmarks, no statistical details (sample sizes, variance, controls for multiple testing), and no ablation that isolates the synthetic generator from benchmark-adjacent distributions. This makes it impossible to assess whether observed gains reflect genuine generalization.

    Authors: We agree that explicit definitions and supporting details are necessary to substantiate the central claim. In the revised manuscript, we will define the Coverage metric in §4, enumerate all downstream benchmarks, report sample sizes, variance, and multiple-testing controls, and add an ablation isolating the synthetic generator. These changes will allow direct evaluation of whether gains reflect generalization beyond benchmark-adjacent distributions. We maintain that the coevolutionary process, rather than explicit optimization, drives the broader coverage, but acknowledge the need for these additions to make the evidence fully transparent. revision: yes

  2. Referee: [§3.2] §3.2 (Task Generation): The synthetic task generator is described without diversity metrics, prompt templates, or controls (e.g., an ablation that replaces it with an independent task source or freezes it after initial generations). Without such tests, the reported Coverage improvements and continual innovation could arise from implicit distributional overlap or closed-loop leakage rather than novel, transferable capabilities.

    Authors: We recognize the value of quantifying diversity and ruling out leakage. The revised §3.2 will include diversity metrics for generated tasks, the full prompt templates, and control ablations that replace the generator with an independent source or freeze it after initial generations. These additions will demonstrate that Coverage gains and innovation arise from the coevolutionary loop rather than distributional overlap. revision: yes

  3. Referee: [§5] §5 (Results on multi-agent selection and Coverage trajectories): The improvements in best-of-N selection and the claim that Coverage increases over time lack baselines that hold the model-merging archive fixed while varying only the task generator (or vice versa). This confounds the contribution of coevolution and weakens the assertion that the framework 'continually innovates.'

    Authors: To isolate the coevolutionary contribution, we will expand §5 with baselines that fix the model-merging archive while varying the task generator, and vice versa. These controls will clarify the joint and separate effects on best-of-N gains and Coverage trajectories, thereby reinforcing the evidence that the framework continually innovates through coevolution. revision: yes

Circularity Check

0 steps flagged

No significant circularity in claimed derivation

full rationale

The paper presents AC/DC as an open-ended coevolution process using model merging for LLMs and synthetic data generation for tasks, with evaluation on external downstream benchmarks explicitly stated as having no optimization involvement. No equations, fitted parameters renamed as predictions, self-citations as load-bearing uniqueness theorems, or ansatzes smuggled via prior work are described in the provided abstract or claims. The central result (broader Coverage without benchmark optimization) is positioned as an empirical outcome of the external loop rather than a definitional or fitted tautology, making the derivation self-contained against held-out benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review is based solely on the abstract; no full methods, equations, or results are available to identify specific free parameters, axioms, or invented entities.

pith-pipeline@v0.9.0 · 5558 in / 1049 out tokens · 31015 ms · 2026-05-10T10:57:33.618653+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

  1. [1]

    The Journal of Open Source Software 2(11) (mar 2017)

    arXiv:2404.01054. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models.arXiv preprint arXiv:2307.10169, 2023. Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, A Feder Cooper, ...

    work page doi:10.21105/joss.00205 2023

  2. [2]

    2018 Symbolic regression and feature construction with GP-GOMEA applied to radiotherapy dose reconstruction of childhood cancer survivors

    August 2024. L B. Soros and Kenneth O. Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. InProc. Int. Conf. on the Sythesis and Simulation of Living Systems (ALIFE), pp. 793–800, Cambridge, MA, 2014. MIT Press. Kenneth O Stanley. Why open-endedness matters.Artificial life, 25(3):232–235, 201...

    work page doi:10.1145/3205455.3205602 2024

  3. [3]

    in-distribution

    explicitly optimizes both diversity and high-quality performance, while maintaining a struc- tured collection (archive) of diverse high-quality solutions with unique behavior characteristics (BCs). Influential algorithms such as MAP-Elites (Mouret & Clune, 2015a; Cully et al., 2015) em- phasize local competition within niches (Lehman & Stanley, 2011b) to ...

    work page 2015

  4. [4]

    Computed pairwise performance differences∆ i =s ′ AC/DC,i −s ′ baseline,i across alln= 8 benchmarks for a given model family (or aggregated across multiple model families)

    work page

  5. [5]

    Generated a bootstrap distribution by resampling the differences{∆ i}n i=1 with replacement 50,000 times, computing the mean difference for each resample

    work page

  6. [6]

    Calculated the bootstrapped mean ¯∆boot and 95% confidence intervals using the percentile method

    work page

  7. [7]

    learnable

    Computed one-tailed p-values to test whether AC/DC shows consistent improvement (i.e., H0 : ¯∆≤0vs.H 1 : ¯∆>0). Lower p-values indicate higher confidence that AC/DC achieves meaningful performance gains. This approach accounts for variance across benchmarks while providing robust statistical evidence for performance improvements. K.2 COVERAGERESULTS K.2.1...

    work page 2026