REVIEW 4 major objections 5 minor 44 references
Evolutionary thoughts: integration of large language models and evolutionary algorithms
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that seeding an evolutionary algorithm with LLM-generated programs and mutating its best individual with the LLM produces more accurate, shorter solutions on list-transformation tasks than plain evolution.
desk verdict A useful engineering contribution and a plausible LLM-boost result, but the missing compute-budget control means the central claim is not yet proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a hybrid evolutionary loop with two LLM interventions. First, the LLM is shown five training instances, produces five separate explanations, and is then asked to synthesize a final problem description; using that description, it generates candidate programs until at least 30 valid ones exist, and those seeds enter the initial population. Second, each generation the current best individual is mutated by the LLM (with up to three retries), and the result is added to the population pool rather than replacing an existing individual, with a reduced application probability to limit diversity loss. The other component is a fast fitness evaluator that represents programs as arrays of typed nodes and executes them with a stack and a switch-case dispatch instead of function calls, allowing runs with population sizes up to 300,000; the CPU multi-core version is reported faster than the GPU version on the hardware used.
What would settle it
Take the 3,000-individual population from an inverse-task run, evaluate every program with both the baseline Python evaluator and the fast compiled evaluator, and compare per-program fitness values and ranks; any mismatch would mean the LLM-versus-baseline comparisons in Table 4 are measuring different landscapes. A second check is to time the fast evaluator against the baseline on identical populations, since the paper claims a computational bottleneck is removed but reports no speed measurement.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that LLM-generated seed individuals plus LLM-guided elitist mutation outperform the unmodified evolutionary algorithm on program-synthesis tasks defined over integer lists, after 1,500 generations of evolution. For the inverse task at population size 3,000, the plain evolutionary algorithm reaches 0.820 training and 0.798 test accuracy, while the LLM-seeded version reaches 1.000 training and 0.997 test accuracy, and adding LLM mutation of the top individual yields 1.000 on both, with the mean program length dropping from 55.0 to 14.3 primitives. The same combination also shortens programs on the sort task, though its accuracy edge at population size 3,000 is small (0.945 training, 0.917 test, versus 0.929/0.894 for plain evolution). The paper further reports that collecting the top individual from several small-population runs and seeding a final run with them (the 'LLM+' setting) reaches perfect accuracy on both inverse and sort across all tested population sizes, with the smallest standard deviations.
Load-bearing premise
The load-bearing premise is that the fast compiled evaluator computes exactly the same fitness values as the baseline Python evaluator, but the paper reports no test of equivalence; if the two evaluators differ in program semantics, the comparisons between LLM-supported and plain evolution are not measuring the same search space.
Editorial extensions
If this is right
- For the inverse task, LLM seeds plus LLM mutation of the top individual reach 1.000 training and test accuracy at population size 3,000, against 0.820/0.798 for plain evolution.
- Ensembling top individuals from several small runs ('LLM+') achieves perfect accuracy on inverse and sort at population sizes as small as 300, which means large populations may be replaceable by repeated smaller runs.
- LLM involvement consistently shortens the discovered programs; on inverse at population size 3,000, mean length falls from 55.0 primitives under plain evolution to 14.3 with seeds plus LLM mutation.
- The fast evaluator makes population sizes up to 300,000 practical in the baseline experiments, where plain evolution reaches perfect accuracy on both inverse and sort.
- The LLM advantage is not universal: on sort at population size 3,000 the LLM-seeded accuracy (0.921 training, 0.866 test) is slightly below plain evolution (0.929/0.894), though the ensemble variant is still the best.
Reading between the lines
- If the advantage generalizes, it likely depends on how describable the target function is: LLM guidance should help most when the LLM can articulate the rule from examples, and an extension would vary task describability while holding population size fixed.
- The description-then-generation seed pipeline can be read as injecting a learned program prior into the population; a testable comparison would replace the LLM with a non-LLM prior, such as random programs biased toward short expressions, to quantify how much of the gain comes from semantic content versus simply better initial diversity.
- The ensemble result suggests that cross-run diversity, not single-run population size, is the main driver of success; an extension would make diversity an explicit selection objective or use multiple LLM descriptions to generate seeds.
- Because the paper does not report wall-clock or cost figures for the LLM calls, the practical value of the method for real workloads remains an open question; a direct extension would compare end-to-end runtime and API cost against plain evolution under equal accuracy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes combining large language models with evolutionary algorithms for program synthesis over list-transformation tasks. The method uses LLM-generated seed individuals (up to 30 valid programs) and LLM-based mutation of the elitist at each generation, alongside a C++/CUDA evaluation framework intended to replace the slower Python DEAP evaluator. Experiments on four tasks (count, max-min, inverse, sort) with population sizes from 300 to 300,000 report training/test accuracy, program length, and tree height, comparing a plain evolutionary algorithm, LLM-seeded runs, and LLM-seeded runs with LLM elitist mutation. The paper claims that LLMs generate superior candidate solutions and that ensembling the top individuals from multiple runs yields more accurate and robust results.
Significance. If the claims hold, the paper provides useful evidence that LLM guidance can focus evolutionary search in program synthesis, producing higher accuracy and shorter programs, and the fast evaluation framework could be a practical engineering contribution. Strengths of the manuscript include: the empirical evaluation uses held-out test splits, avoiding internal circularity; standard deviations are reported for the main repeated-run tables; the code repositories are made available; and the ensembling-of-runs idea is a sensible way to combine diversity with small populations. The central empirical claim is plausible but currently under-supported because the LLM-aided pipeline differs from the baseline in ways beyond semantic guidance, and several empirical controls are missing.
major comments (4)
- [3.2.2 and 3.2.4, Tables 2-4] The LLM-aided runs differ from the baseline in the number of evaluated individuals and retries. The LLM seed procedure is repeated until at least 30 valid functions are produced, and LLM mutation of the elitist adds an extra individual to the population pool with up to three retries, whereas the pure-EA baseline in Table 1 receives no additional seeds and no retries. Consequently, the improved accuracy and shorter programs in Tables 2-4 cannot be uniquely attributed to semantic guidance from the LLM; they may instead result from the larger exploration budget. Please add ablation controls that keep the number of individuals and evaluations constant across conditions, for example 30 random extra seeds of the same maximum length, and an elitist mutated by the same number of random subtree mutations with the same retry allowance.
- [5.1, Table 5] The starred ensemble rows in Table 5 are single runs without error bars, yet the text states that "in all cases, when using the seed functions, the accuracy is largely improved" and claims robustness. Moreover, the sortedEA 3000 baseline row in Table 5 reports 0.977 (0.05) training and 0.950 (0.10) testing, whereas Table 1 reports 0.929 (0.14) and 0.894 (0.18) for the same configuration; this discrepancy directly affects the comparison with sortedLLM 3000 (0.921/0.866). Please reconcile these numbers and report multiple ensemble runs with variance to support the robustness claim.
- [3.4] The fast evaluation framework is a stated contribution, but no equivalence test between the C++/CUDA evaluator and the Python DEAP evaluator is reported, and no runtime measurements are provided. If the compiled evaluator handles loop bounds, register reads, or other semantics differently from the Python interpreter, the fitness landscape for LLM-aided runs could differ from that of the baseline runs. Please report a semantic equivalence check on a representative set of programs (including edge cases for loops and registers) and benchmark wall-clock evaluation times for the population sizes used.
- [4.2, Tables 1-4] All conclusions rest on ten runs with no significance tests or effect sizes, and several reported differences have overlapping standard deviations (for example, inverse at population size 1000: EA 0.791 (0.22) versus LLM 0.845 (0.24); sorted at population size 3000: EA 0.929 (0.14) versus LLM 0.921 (0.13)). Please add paired significance tests across runs or report effect sizes to support the claim that LLM guidance improves outcomes.
minor comments (5)
- [Abstract] The sentence "However, EAs explore a vast search space when applied to complex problems" appears twice, which creates a redundant contrast with the preceding sentence.
- [Listing 7] The comment "The fist use case 83" should read "The first use case 83".
- [References] Reference [42] contains a typo: "large anguage models" should be "large language models".
- [Table 5 caption] The caption says "Mean accuracy results ... using the evolutionary algorithm" but the table also includes LLM and LLM+ rows; please clarify the caption to reflect the full set of conditions.
- [3.2.4] The statement that LLM mutations "do not substitute existing individuals" but are added to the population pool is not followed by a description of how the population is trimmed to size; please state the exact number of individuals evaluated per generation in each condition.
Circularity Check
No significant circularity: the LLM-benefit claim is evaluated on held-out test accuracy, with no fitted parameter or self-citation chain that would force the result.
full rationale
The paper's central claim is empirical: LLM-generated seeds and LLM-guided mutations improve the top evolved program's training and test accuracy (Tables 2-4) relative to a plain evolutionary algorithm (Table 1). Fitness is computed as average accuracy against ground-truth input-output pairs, and the reported test metrics come from held-out examples, not from any quantity the LLM was fitted to. No equation in the paper defines the target in terms of the LLM output or vice versa, and there are no fitted parameters renamed as predictions. The authors do not invoke a self-citation chain or a uniqueness theorem to force their choice of approach. The main concerns are experimental rather than circular: the C++/CUDA evaluator is not shown to be semantically identical to the DEAP baseline, and the LLM path receives extra initial seeds and retries, so part of the observed gain may reflect additional search budget. Those are legitimate validity risks, but they do not make the derivation equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- Fitness scaling constant =
50
- Length slack =
25 primitives
- Number of LLM seed individuals =
30
assumptions (4)
- domain assumption LLM-generated programs are valid and useful
- domain assumption Training accuracy proxies test accuracy
- domain assumption C++ evaluator semantics match Python DEAP semantics
- domain assumption Ten runs and 1500 generations are sufficient
Cite this review
Pith. "Pith review of Evolutionary thoughts: integration of large language models and evolutionary algorithms." pith.science (2026). https://pith.science/paper/WYDKXX5F
@misc{pith2026250505756,
author = {Pith},
title = {Pith review of: Evolutionary thoughts: integration of large language models and evolutionary algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/WYDKXX5F}},
note = {Machine review of arXiv:2505.05756}
}
read the original abstract
Large Language Models (LLMs) have unveiled remarkable capabilities in understanding and generating both natural language and code, but LLM reasoning is prone to hallucination and struggle with complex, novel scenarios, often getting stuck on partial or incorrect solutions. However, the inherent ability of Evolutionary Algorithms (EAs) to explore extensive and complex search spaces makes them particularly effective in scenarios where traditional optimization methodologies may falter. However, EAs explore a vast search space when applied to complex problems. To address the computational bottleneck of evaluating large populations, particularly crucial for complex evolutionary tasks, we introduce a highly efficient evaluation framework. This implementation maintains compatibility with existing primitive definitions, ensuring the generation of valid individuals. Using LLMs, we propose an enhanced evolutionary search strategy that enables a more focused exploration of expansive solution spaces. LLMs facilitate the generation of superior candidate solutions, as evidenced by empirical results demonstrating their efficacy in producing improved outcomes.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023
arXiv 2023
-
[2]
Program synthesis with large language models
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021
arXiv 2021
-
[3]
Reducing bias and inefficiency in the selection algorithm
James E Baker et al. Reducing bias and inefficiency in the selection algorithm. In Proceedings of the second international conference on genetic algorithms , volume 206, pages 14–21, 1987
work page 1987
-
[4]
Language models are few-shot learn- ers
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems , 33:1877–1901, 2020
1901
-
[5]
Algorithmic information theory
Gregory J Chaitin. Algorithmic information theory. IBM journal of research and development , 21(4):350–359, 1977
1977
-
[6]
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. 13 Task PS Training Testing Length Height inverseEA 300 0.749 (0.21) 0.722 (0.23) 84.50 (68.19) 12.10 (5.89) inverseEA* 300 1.000 1.000 15 5 inverseEA 1000...
arXiv 2022
-
[7]
Arc prize 2024: Technical report
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604 , 2024
arXiv 2024
-
[8]
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023
2023
Show all 44 references
-
[9]
An analysis of the behavior of a class of genetic adaptive systems
Kenneth Alan De Jong. An analysis of the behavior of a class of genetic adaptive systems. University of Michigan, 1975
1975
-
[10]
Deap: A python framework for evolutionary algorithms
Fran¸ cois-Michel De Rainville, F´ elix-Antoine Fortin, Marc-Andr´ e Gardner, Marc Parizeau, and Christian Gagn´ e. Deap: A python framework for evolutionary algorithms. In Proceedings of the 14th annual conference companion on Genetic and evolutionary computation , pages 85–92, 2012
2012
-
[11]
Multi-objective optimisation using evolutionary algorithms: an introduction
Kalyanmoy Deb. Multi-objective optimisation using evolutionary algorithms: an introduction. In Multi- objective evolutionary optimisation for product design and manufacturing , pages 3–34. Springer, 2011
2011
-
[12]
Introduction to evolutionary computing
Agoston E Eiben and James E Smith. Introduction to evolutionary computing . Springer, 2015
2015
-
[13]
Preventing premature convergence in genetic algorithms by preventing incest
Larry J Eshelman. Preventing premature convergence in genetic algorithms by preventing incest. In Proceedings of Fourth International Conference on Genetic Algorithms, 1991 , 1991
1991
-
[14]
Genetic algorithms: principles of natural selection applied to computation
Stephanie Forrest. Genetic algorithms: principles of natural selection applied to computation. Science, 261(5123):872–878, 1993
1993
-
[15]
Deap: Evolutionary algorithms made easy.The Journal of Machine Learning Research, 13(1):2171–2175, 2012
F´ elix-Antoine Fortin, Fran¸ cois-Michel De Rainville, Marc-Andr´ e Gardner Gardner, Marc Parizeau, and Christian Gagn´ e. Deap: Evolutionary algorithms made easy.The Journal of Machine Learning Research, 13(1):2171–2175, 2012
2012
-
[16]
Incoder: A generative model for code infilling and synthesis
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 , 2022
2022 arXiv
-
[17]
Goldberg
David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Longman Publishing Co., Inc., USA, 1st edition, 1989
1989
-
[18]
Inductive programming meets the real world
Sumit Gulwani, Jos´ e Hern´ andez-Orallo, Emanuel Kitzelmann, Stephen H Muggleton, Ute Schmid, and Benjamin Zorn. Inductive programming meets the real world. Communications of the ACM, 58(11):90– 99, 2015
2015
-
[19]
Training compute- optimal large language models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute- optimal large language models. arXiv preprint arXiv:2203.15556 , 2022
2022 arXiv
-
[20]
J.H. Holland. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence . Complex Adaptive Systems. MIT Press, 1992
1992
-
[21]
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024
2024 arXiv
-
[22]
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199–22213, 2022
2022
-
[23]
Genetic programming: On the programming of computers by means of natural selection (complex adaptive systems)
John R Koza. Genetic programming: On the programming of computers by means of natural selection (complex adaptive systems). A Bradford Book, 1:18, 1993
1993
-
[24]
Genetic programming and data structures
William Benjamin Langdon. Genetic programming and data structures. University of London, University College London (United Kingdom), 1996. 15
1996
-
[25]
Pre-trained language models for interactive decision-making
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Aky¨ urek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems , 35:31199–31212, 2022
2022
-
[26]
Self-refine: Iterative refinement with self- feedback
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems , 36:46534–46594, 2023
2023
-
[27]
Niching methods for genetic algorithms
Samir W Mahfoud. Niching methods for genetic algorithms. University of Illinois at Urbana-Champaign, 1995
1995
-
[28]
Designing neural networks using genetic algorithms
Geoffrey F Miller, Peter M Todd, and Shailesh U Hegde. Designing neural networks using genetic algorithms. In ICGA, volume 89, pages 379–384, 1989
1989
-
[29]
An introduction to genetic algorithms
Melanie Mitchell. An introduction to genetic algorithms . MIT press, 1998
1998
-
[30]
Codegen: An open large language model for code with multi-turn program synthesis
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022
2022 arXiv
-
[31]
Genetic set recombination
Nicholas J Radcliffe. Genetic set recombination. In Foundations of genetic algorithms , volume 2, pages 203–219. Elsevier, 1993
1993
-
[32]
Scaling language models: Methods, analysis & insights from training gopher
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021
2021 arXiv
-
[33]
Multitask prompted training enables zero- shot task generalization
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero- shot task generalization. arXiv preprint arXiv:2110.08207 , 2021
-
[34]
Distilling free-form natural laws from experimental data
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81–85, 2009
2009
-
[35]
Crossover or mutation? In Foundations of genetic algorithms , volume 2, pages 221–237
William M Spears. Crossover or mutation? In Foundations of genetic algorithms , volume 2, pages 221–237. Elsevier, 1993
1993
-
[36]
Lamda: Language models for dialog applica- tions
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.08239 , 2022
2022 arXiv
-
[37]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023
2023 arXiv
-
[38]
The nature of statistical learning theory
Vladimir N Vapnik. The nature of statistical learning theory . Springer-verlag New York, 1995
1995
-
[39]
Statistical learning theory
Vladimir N Vapnik. Statistical learning theory. Wiley, 1998
1998
-
[40]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022
2022 arXiv
-
[41]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[42]
Griffiths, Yuan Cao, and Kathik Narasimham
Shunyu Yao, Dian Yu, Jeffery Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Kathik Narasimham. Tree of thoughts: Deliberate problem solving with large anguage models. arXiv preprint arXiv:2305.10601v2, 2023. 16
2023 arXiv
-
[43]
Evolving artificial neural networks
Xin Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447, 1999
1999
-
[44]
Fine-tuning language models from human preferences
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 17 Appendix A - Example functions generated by the best hyper...
1909 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.