Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that behaviour-space metrics explain why a 1+1 elitist LLaMEA variant using simplify and random-new mutation prompts outperforms five rivals on BBOB functions.

desk verdict A credible, well-scoped empirical study of LLM-driven algorithm discovery that combines behavior metrics and code-evolution graphs, but its headline ranking rests on five runs and needs statistical support before it can be called a demonstration. read the letter →

arxiv 2507.03605 v1 pith:BVIYQ3AD submitted 2025-07-04 cs.NE cs.AI

Niki van Stein , Haoran Yin , Anna V. Kononova , Thomas Bäck , Gabriela Ochoa This is my paper
classification cs.NEcs.AI
keywords automatedalgorithmdesignlargelanguagemodelsmeta-heuristicdiscoverybehaviourspaceanalysissearchtrajectorynetworkscodeevolutiongraphsblack-boxoptimisationexploration-exploitationbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why some automatically generated optimisation heuristics beat others, and to give a method for seeing the difference. Working with the LLaMEA framework and a GPT-based LLM, the authors evolve algorithms on 10 noiseless BBOB functions and record full search traces, then convert each trace into behavioural metrics for exploration, exploitation, convergence, and stagnation. They compare six LLaMEA configurations with different mutation prompts and evolutionary parameters. Their central finding is that the elitist (1+1) configuration using both a refine-and-simplify prompt and a generate-new-random-algorithm prompt outperforms the other five in area over the convergence curve, and that its advantage is visible in behaviour space as more exploitation, faster convergence, and less stagnation. A sympathetic reader would care because this offers a diagnostic view of LLM-driven algorithm design, beyond which algorithm wins.

What carries the argument

The load-bearing machinery is a set of scalar behaviour metrics computed from complete evaluation traces, plus two network visualisations built on them. The metrics include exploration percentage, average nearest-neighbour distance, coverage dispersion, distance to best, intensification ratio, convergence rate, success rate, longest no-improvement streak, and last-improvement fraction; they position any run as a point in a low-dimensional behaviour space. Search Trajectory Networks partition that behaviour space into hypercubes and connect successive states to show whether a configuration drifts, branches, or follows one improving path. Code Evolution Graphs link each generated algorithm to its parent and overlay static code features such as token count and AST complexity. Together these tools carry the argument that the measured search dynamics, not just final scores, distinguish the winning configuration.

What would settle it

Re-run the same six LLaMEA configurations on the same 10 BBOB functions with a substantially larger number of independent seeds, say 30, and compare mean AOCC with confidence intervals; if LLaMEA-4 is no longer the top-ranked configuration, the paper's central performance claim is refuted. A second check would repeat the experiment in higher dimension or with a different LLM and see whether the same behaviour-space signature still tracks the best performer.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that behaviour-space analysis can explain the performance gap between LLM-generated meta-heuristics. Using the six LLaMEA configurations, the paper reports that LLaMEA-4 — the variant that combines 'refine and simplify' with 'generate a new algorithm' mutation prompts and runs a strict 1+1 elitist evolution strategy — achieves the best and most stable Area Over the Convergence Curve on the test instances. The trace metrics show its runs have lower exploration percentage, smaller nearest-neighbour distances, faster average convergence, higher intensification near the best solution, and less final stagnation than the population-based variants. Its search trajectory networks are compact elongated paths with few wasted branches, while its code evolution graphs show performance improving without code bloat. The paper presents this as evidence that an exploration-exploitation balance, protected by elitism, is what makes LLM-designed algorithms succeed.

Load-bearing premise

The ordering of the six configurations depends on five independent runs per configuration being representative, and the paper itself cautions that this limited run count means conclusions are drawn with care; if additional runs reshuffle the ranking, the claim that LLaMEA-4 is best would fall even though the behaviour-space method could remain sound.

Editorial extensions

If this is right

  • Using two contrasting mutation prompts — refine-and-simplify alongside generate-new — is better than either prompt alone, because it keeps both exploitation and exploration pressure in the search over algorithms.
  • An elitist (1+1) selection scheme is valuable when evaluating algorithms is expensive, since it guarantees the best discovered algorithm is never discarded.
  • Behaviour metrics such as last-improvement fraction and no-improvement streak can diagnose why a design run failed — early stagnation or excessive exploration — without waiting for final performance.
  • Keeping generated code simple is not just a stylistic preference; the Code Evolution Graphs associate lower token counts with continued performance gains and better generalisation.
  • The same trace-based analysis tools developed for hand-designed meta-heuristics transfer to LLM-generated algorithms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: the behaviour metrics could be computed online during evolution and fed back as an adaptive fitness signal, letting the system increase exploration when stagnation metrics climb.
  • The behaviour-space footprints may transfer across problem suites: if an algorithm's metric profile predicts its AOCC on BBOB, the same profile could be used as a cheap pre-screen on new problems before full evaluation.
  • Because the paper uses one LLM (GPT o4-mini), its ranking is entangled with that model's coding style; testing with other LLMs could reframe which prompt strategy is best even if the visualisation methodology survives.
  • A cautious extension: with only five runs per configuration, the headline ranking should be read as provisional; the behaviour-space explanation is the more robust contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents an empirical study of six LLaMEA configurations for LLM-driven meta-heuristic discovery on 10 BBOB functions in 5D. The authors log optimisation traces, compute behaviour metrics (exploration, exploitation, convergence, stagnation), and visualize the results using parallel coordinates, Code Evolution Graphs, and Search Trajectory Networks. The central claim is that LLaMEA-4, which uses both simplification and random-perturbation prompts in a (1+1) elitist strategy, achieves the highest Area Over the Convergence Curve, and that behaviour-space analysis explains this advantage through more intensive exploitation, faster convergence, and less stagnation. The paper also presents qualitative comparisons of code evolution and search-trajectory structure across the six variants.

Significance. If the central ranking is robust, the paper makes a useful contribution by extending behaviour-space analysis, originally developed for human-designed metaheuristics, to LLM-generated algorithms. The combination of dynamic trace metrics with static code analysis (CEGs) is a genuinely novel angle, and the authors ship an implementation of the metrics in a public repository, which supports reproducibility. The conclusions are falsifiable and the methodology is transferable to other LLM-based algorithm discovery frameworks. However, the significance is currently limited by the statistical fragility of the headline ranking and by the qualitative nature of several behavioural claims; the paper's own limitations section acknowledges that only five runs were performed. The contribution is more of a case study than a definitive methodological validation, but it is a valuable template for future work.

major comments (4)
  1. [Section 4.1, Figure 1] The central claim that LLaMEA-4 achieves the highest AOCC is not supported by any statistical test. The mean AOCC curves in Figure 1 have overlapping 95% confidence intervals for most variants, and Section 4.1 itself states that most variants show a relatively large variation between runs except for LLaMEA-4. Because the behaviour-space explanation in Sections 4.3 and 5 compares higher-performing and lower-performing groups based on this ranking, an unstable ranking would invalidate the interpretation. Please add a formal comparison (e.g., paired Wilcoxon signed-rank or bootstrap on per-run aggregated AOCC values, with multiple-testing correction) and report per-function AOCC tables.
  2. [Section 3.2] The hand-chosen thresholds used to define the behaviour metrics are not justified: the intensification radius (10% of the search range), the chunk size K in the exploration percentage, and the STN hypercube dimensions 0.01 and 0.1 are selected without sensitivity analysis. The qualitative conclusions of Section 4.3, such as the distinction between bushy and elongated STNs, may depend on these choices. Please add a sensitivity study or otherwise demonstrate that the main ranking and behavioural contrasts are stable with respect to these parameters.
  3. [Section 3.1 and Section 4.1] The paper does not enumerate the 10 BBOB functions used for evaluation, although Section 3.1 says they were chosen to cover a range of landscapes. This makes it impossible to assess whether the aggregate ranking is dominated by a few functions; Figure 4 (right) shows different behaviour profiles for different functions, which further motivates a per-function breakdown. Please list the BBOB function IDs or names and report the AOCC values per function for each configuration.
  4. [Section 4.3 and Discussion item (1)] The behavioural explanation is largely qualitative. The claim that higher-performing algorithms exhibit more intensive exploitation, faster convergence, and less stagnation is based on visual inspection of parallel coordinate plots and STNs, with no quantitative comparison across performance quartiles. Moreover, the comparison of LLaMEA-3 and LLaMEA-4 confounds the mutation-prompt strategy with the evolutionary selection scheme (4/12 population vs. 1+1 elitist); since LLaMEA-6 is also 1+1 elitist but has wide confidence intervals, the paper does not establish that elitism is the decisive factor. Please quantify the metric differences between performance groups and address the confound explicitly.
minor comments (5)
  1. [Section 3.2] The display equation for D(S) has a garbled summation range: the expression "p<q xp,xq∈S" should be rewritten as a proper double sum over distinct pairs.
  2. [Figure 1] The right subplot does not label the y-axis; it should clearly state that the plotted quantity is the AOCC of the final generated algorithms evaluated on the test instances.
  3. [Section 4.3] There is a typo: "On the other and" should read "On the other hand".
  4. [Sections 2.2 and 3.3] The phrases "prefixed lower numerical precision" and "prefixed dimension" are ambiguous; please specify whether the STN hypercube side lengths are given in normalized units or in the raw units of the behaviour metrics.
  5. [Reference [1]] The repository link is anonymized in this version; in the final published version, the link should be made permanent and versioned so that the metric implementations are fully reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central ranking is an empirical comparison on held-out BBOB test instances, with self-citations serving only as tooling.

full rationale

The paper's central claim is that LLaMEA-4 achieves the highest AOCC among six configurations. This is an empirical result, not a derivation: algorithms are evolved using AOCC on training instances (BBOB instances 1–5), while the reported final ranking is based on AOCC on held-out test instances (instances 6–15), as stated in Section 3.1: 'we use the BLADE experiment setup with training instances to evolve algorithms, then report performance on test instances for final evaluation.' No fitted parameter is recycled as a prediction, and no quantity is defined in terms of the outcome it is supposed to explain. The behaviour metrics (exploration, exploitation, convergence, stagnation) are computed post hoc from logged traces and are used for descriptive interpretation, not as fitness inputs. Self-citations to LLaMEA, BLADE, CEG, and the metrics repository are references to the authors' own tools and prior framework, but the load-bearing evidence for the ranking is an independent external benchmark (BBOB) with a train/test split; no self-citation is invoked as a uniqueness theorem or as a substitute for experimental support. The acknowledged limitation that only 5 runs were performed ('The still limited number of 5 runs also means that the conclusions of these work are made with caution') is a statistical robustness concern, not circularity. No circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No free-parameter fitting is used to produce the central performance comparison, but several behavioural metrics used to explain that comparison depend on hand-chosen thresholds (intensification radius, STN resolution, chunk size K), and the whole analysis assumes the metrics are valid proxies and that five runs represent each variant. No new physical or mathematical entities are introduced; behaviour space is an analytical construct over logged traces, not an independent postulate.

free parameters (3)
  • Intensification ratio radius = 10% of the search range
    Section 3.2 defines a point as intensifying if it lies within 10% of the search range of the best-so-far solution; the threshold is hand-chosen and directly shapes the exploitation measures used to explain LLaMEA-4's success.
  • STN hypercube side length = 0.01 and 0.1
    Section 3.3 groups behaviour vectors into hypercubes of side 0.01, and optionally 0.1 for a coarse view; the resulting network shape and component counts depend on this choice.
  • Exploration-percentage chunk size K = not specified
    The metric definition in Section 3.2 partitions traces into chunks of size K to compute exploration percentage, but K is never stated; it affects the exploration and exploitation values central to the behavioural explanation.
assumptions (4)
  • domain assumption The defined behaviour metrics faithfully measure exploration, exploitation, convergence and stagnation for LLM-generated continuous optimizers.
    Section 3.2 introduces the metrics as proxies and the paper's explanations of why LLaMEA-4 wins rely on these proxies being meaningful. Some are adapted from prior work, but their validity for LLM-generated optimizers is assumed.
  • domain assumption Aggregating mean AOCC across the ten BBOB functions into a single scalar is a fair fitness signal for ranking algorithms.
    Section 3.1 evaluates each algorithm as mean performance across training instances and functions; no normalization of function scales is described.
  • domain assumption Five independent runs per configuration are representative of each variant's distribution.
    Section 4.1 and limitation (4) acknowledge only five runs due to API and computational expense; all conclusions about ranking and dynamics inherit this assumption.
  • ad hoc to paper Behaviour-space hypercube partitioning and the five selected metrics capture the relevant search trajectory topology.
    Section 3.3 chooses hypercube side lengths 0.01 and 0.1 and the five least-correlated metrics without independent justification; the STN visual conclusions depend on these choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery." pith.science (2026). https://pith.science/paper/BVIYQ3AD

@misc{pith2026250703605,
  author       = {Pith},
  title        = {Pith review of: Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BVIYQ3AD}},
  note         = {Machine review of arXiv:2507.03605}
}
read the original abstract

We investigate the behaviour space of meta-heuristic optimisation algorithms automatically generated by Large Language Model driven algorithm discovery methods. Using the Large Language Evolutionary Algorithm (LLaMEA) framework with a GPT o4-mini LLM, we iteratively evolve black-box optimisation heuristics, evaluated on 10 functions from the BBOB benchmark suite. Six LLaMEA variants, featuring different mutation prompt strategies, are compared and analysed. We log dynamic behavioural metrics including exploration, exploitation, convergence and stagnation measures, for each run, and analyse these via visual projections and network-based representations. Our analysis combines behaviour-based projections, Code Evolution Graphs built from static code features, performance convergence curves, and behaviour-based Search Trajectory Networks. The results reveal clear differences in search dynamics and algorithm structures across LLaMEA configurations. Notably, the variant that employs both a code simplification prompt and a random perturbation prompt in a 1+1 elitist evolution strategy, achieved the best performance, with the highest Area Over the Convergence Curve. Behaviour-space visualisations show that higher-performing algorithms exhibit more intensive exploitation behaviour and faster convergence with less stagnation. Our findings demonstrate how behaviour-space analysis can explain why certain LLM-designed heuristics outperform others and how LLM-driven algorithm discovery navigates the open-ended and complex search space of algorithms. These findings provide insights to guide the future design of adaptive LLM-driven algorithm generators.

Figures

Figures reproduced from arXiv: 2507.03605 by the authors.

Figure 1
Figure 1. Aggregated results of the 5 runs per LLaMEA variant by computing the mean best AOCC at each evaluation step (left subplot). A higher AOCC means the generated algorithm finds good optima faster on average. 95% confidence intervals are visualized as shaded area. On the right the AOCC distribution of the 5 final generated algorithms per LLaMEA configuration, evaluated on the 10 test instances [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 2
Figure 2. CEGs for the BBOB runs under different LLaMEA configurations [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Pearson [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Behaviour profile over all generated algorithms (left) and the best [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Search trajectory networks for the 6 LLaMEA variants with partition [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Search trajectory networks for the 6 LLaMEA variants with partition [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving

    cs.NE 2025-09 conditional novelty 4.0 of 10

    A literature survey that classifies LLM-based optimization research into modeling and solving, with solving divided into LLMs as optimizers, low-level components, and high-level managers.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://doi.org/10.5281/zenodo.15675581, https://doi.org/10.5281/zenodo.15675581

    Anonymous: Behaviour metrics for llm-driven automated algo- rithm design (Jun 2025). https://doi.org/10.5281/zenodo.15675581, https://doi.org/10.5281/zenodo.15675581

    work page doi:10.5281/zenodo.15675581 2025

  2. [2]

    arXiv preprint arXiv:2505.15741 (2025)

    Chauhan, D., Dutta, B., Bala, I., van Stein, N., Bäck, T., Yadav, A.: Evolution- ary computation and large language models: A survey of methods, synergies, and applications. arXiv preprint arXiv:2505.15741 (2025)

    arXiv 2025

  3. [3]

    ACM computing surveys (CSUR)45(3), 1–33 (2013)

    Črepinšek, M., Liu, S.H., Mernik, M.: Exploration and exploitation in evolutionary algorithms: A survey. ACM computing surveys (CSUR)45(3), 1–33 (2013)

    work page 2013

  4. [4]

    In: International Conference on Machine Learning (ICML) (2024), https://arxiv.org/abs/2401.02051

    Fei, L., Tong, X., Yuan, M., Lin, X., Luo, F., Wang, Z., Lu, Z., Zhang, Q.: Evo- lution of heuristics: Towards efficient automatic algorithm design using large lan- guage model. In: International Conference on Machine Learning (ICML) (2024), https://arxiv.org/abs/2401.02051

    arXiv 2024

  5. [5]

    IEEE Transactions on Evolutionary Com- putation 26(6), 1293–1305 (2022)

    Hansen, N., Auger, A., Brockhoff, D., Tušar, T.: Anytime performance assessment in blackbox optimization benchmarking. IEEE Transactions on Evolutionary Com- putation 26(6), 1293–1305 (2022)

    work page 2022

  6. [6]

    Research Report RR-6829, INRIA (2009), https://hal.inria.fr/inria-00362633

    Hansen, N., Finck, S., Ros, R., Auger, A.: Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829, INRIA (2009), https://hal.inria.fr/inria-00362633

    work page 2009

  7. [7]

    IEEE Transactions on Evolutionary Computation 20(2), 316–321 (2016)

    He, J., Lin, G.: Average convergence rate of evolutionary algorithms. IEEE Transactions on Evolutionary Computation 20(2), 316–321 (2016). https://doi.org/10.1109/TEVC.2015.2444793

    work page arXiv 2016

  8. [8]

    Communications of the ACM55(2), 70–80 (2012)

    Hoos, H.H.: Programming by optimization. Communications of the ACM55(2), 70–80 (2012)

    work page 2012

Show all 37 references
  1. [9]

    In: European Conference on Genetic Programming (Part of EvoStar)

    Hu, T., Ochoa, G., Banzhaf, W.: Phenotype search trajectory networks for linear genetic programming. In: European Conference on Genetic Programming (Part of EvoStar). pp. 52–67. Springer (2023)

    2023

  2. [10]

    IEEE Transactions on Evolutionary Computation 19(1), 118–135 (2015)

    Langdon, W.B., Harman, M.: Optimizing existing software with genetic program- ming. IEEE Transactions on Evolutionary Computation 19(1), 118–135 (2015). https://doi.org/10.1109/TEVC.2013.2281544

    2015

  3. [11]

    arXiv preprint arXiv:2410.14716 (2024)

    Liu, F., Yao, Y., Guo, P., Yang, Z., Zhao, Z., Lin, X., Tong, X., Yuan, M., Lu, Z., Wang, Z., et al.: A systematic survey on large language models for algorithm design. arXiv preprint arXiv:2410.14716 (2024)

    2024

  4. [12]

    In: Proceedings of the 13th annual conference on Genetic and evolutionary computation

    Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., Rudolph, G.: Exploratory landscape analysis. In: Proceedings of the 13th annual conference on Genetic and evolutionary computation. pp. 829–836 (2011)

    2011

  5. [13]

    In: Genetic Programming

    Nadizar, G., Rusin, F., Medvet, E., Ochoa, G.: The role of stepping stones in map-elites: Insights from search trajectory networks. In: Genetic Programming. pp. 224–239. Springer Nature Switzerland, Cham (2025)

    2025

  6. [14]

    arXiv preprint arXiv:2410.14573 (2024)

    Nezami, N., Anahideh, H.: Building trust in black-box optimization: A compre- hensive framework for explainability. arXiv preprint arXiv:2410.14573 (2024)

    2024 arXiv

  7. [15]

    In: Proc

    de Nobel, J., Vermetten, D., Wang, H., Doerr, C., Bäck, T.: Tun- ing as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules. In: Proc. of Genetic and Evolu- tionary Computation Conference (GECCO’21, Companion material). pp. 1375–1384. ...

    2021

  8. [16]

    de Nobel, J., Ye, F., Vermetten, D., Wang, H., Doerr, C., Bäck, T.: Iohex- perimenter: Benchmarking platform for iterative optimization heuristics. Evol. 18 N. van Stein et al. Comput. 32(3), 205–210 (2024). https://doi.org/10.1162/EVCO\_A\_00342, https://doi.org/10.1162/evco_a_00342

    2024 doi

  9. [17]

    Applied Soft Computing 109, 107492 (2021)

    Ochoa, G., Malan, K.M., Blum, C.: Search trajectory networks: A tool for analysing and visualising the behaviour of metaheuristics. Applied Soft Computing 109, 107492 (2021). https://doi.org/https://doi.org/10.1016/j.asoc.2021.107492, https://www.sciencedirect.com/science/arti...

    2021

  10. [18]

    Recent advances in the theory and ap- plication of fitness landscapes pp

    Ochoa, G., Verel, S., Daolio, F., Tomassini, M.: Local optima networks: A new model of combinatorial fitness landscapes. Recent advances in the theory and ap- plication of fitness landscapes pp. 233–262 (2014)

    2014

  11. [19]

    arXiv preprint arXiv:2407.11511 (2024)

    Plaat, A., Wong, A., Verberne, S., Broekens, J., van Stein, N., Back, T.: Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511 (2024)

    2024

  12. [20]

    In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

    Pluhacek, M., Kazikova, A., Kadavy, T., Viktorin, A., Senkerik, R.: Leveraging large language models for the generation of novel metaheuristic optimization algo- rithms. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation. pp. 1812–1820 (2023)

    2023

  13. [21]

    In: International Conference on Automated Machine Learning

    Pulatov, D., Anastacio, M., Kotthoff, L., Hoos, H.: Opening the black box: Au- tomated software analysis for algorithm selection. In: International Conference on Automated Machine Learning. pp. 6–1. PMLR (2022)

    2022

  14. [22]

    In: 2016 IEEE Symposium Series on Computational Intelligence (SSCI)

    van Rijn, S., Wang, H., van Leeuwen, M., Bäck, T.: Evolving the structure of evo- lution strategies. In: 2016 IEEE Symposium Series on Computational Intelligence (SSCI). pp. 1–8 (2016). https://doi.org/10.1109/SSCI.2016.7850138

    2016

  15. [23]

    Nature 625, 468–475 (01 2024)

    Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M.P., Dupont, E., Ruiz, F.J., Ellenberg, J.S., Wang, P., Fawzi, O., Kohli, P., Fawzi, A.: Mathematical discoveries from program search with large language models. Nature 625, 468–475 (01 2024)

    2024

  16. [24]

    In: International Conference on the Applications of Evolutionary Computa- tion (Part of EvoStar)

    Sarti, S., Adair, J., Ochoa, G.: Neuroevolution trajectory networks of the behaviour space. In: International Conference on the Applications of Evolutionary Computa- tion (Part of EvoStar). pp. 685–703. Springer (2022)

    2022

  17. [25]

    IEEE Transactions on Evolutionary Computation pp

    van Stein, N., Bäck, T.: Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Transactions on Evolutionary Computation pp. 1–1 (2024). https://doi.org/10.1109/TEVC.2024.3497793

    2024

  18. [26]

    arXiv preprint arXiv:2503.16668 (2025)

    van Stein, N., Kononova, A.V., Kotthoff, L., Bäck, T.: Code evolution graphs: Understanding large language model driven design of algorithms. arXiv preprint arXiv:2503.16668 (2025)

    2025 arXiv

  19. [27]

    arXiv preprint arXiv:2504.20183 (2025)

    van Stein, N., Kononova, A.V., Yin, H., Bäck, T.: Blade: Benchmark suite for llm- driven automated design and evolution of iterative optimisation heuristics. arXiv preprint arXiv:2504.20183 (2025)

    2025 arXiv

  20. [28]

    ACM Transactions on Evolutionary Learning (2024)

    van Stein, N., Vermetten, D., Bäck, T.: In-the-loop hyper-parameter optimization for llm-based automated design of heuristics. ACM Transactions on Evolutionary Learning (2024)

    2024

  21. [29]

    vanStein,N.,Vermetten,D.,Kononova,A.V.,Bäck,T.:Explainablebenchmarking for iterative optimization heuristics (2024), arXiv:2401.17842

    2024 arXiv

  22. [30]

    In: Handbook of metaheuristics, pp

    Stützle, T., López-Ibáñez, M.: Automated design of metaheuristic algorithms. In: Handbook of metaheuristics, pp. 541–579. Springer (2018)

    2018

  23. [31]

    Scientific Reports14(1), 17863 (2024)

    Subburaj, B., Maheswari, J.U., Ibrahim, S.S., Kavitha, M.S.: Population diversity control based differential evolution algorithm using fuzzy system for noisy multi- objective optimization problems. Scientific Reports14(1), 17863 (2024)

    2024

  24. [32]

    In: Computer Graphics Forum

    Telea, A., Auber, D.: Code flows: Visualizing structural evolution of source code. In: Computer Graphics Forum. vol. 27, pp. 831–838. Wiley Online Library (2008) Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery 19

    2008

  25. [33]

    In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar)

    Thomson, S.L., Renau, Q., Vermetten, D., Hart, E., Stein, N.v., Kononova, A.V.: Stalling in space: Attractor analysis for any algorithm. In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar). pp. 510–526. Springer (2025)

    2025

  26. [34]

    In: Proceedings of the Genetic and Evolutionary Computa- tion Conference

    Vermetten, D., Caraffini, F., Kononova, A.V., Bäck, T.: Modular differen- tial evolution. In: Proceedings of the Genetic and Evolutionary Computa- tion Conference. p. 864–872. GECCO ’23, Association for Computing Ma- chinery, New York, NY, USA (2023). https://doi.org/10.1145/3...

    2023

  27. [35]

    arXiv preprint arXiv:2402.01145 (2024)

    Ye, H., Wang, J., Cao, Z., Berto, F., Hua, C., Kim, H., Park, J., Song, G.: Reevo: Large language models as hyper-heuristics with reflective evolution. arXiv preprint arXiv:2402.01145 (2024)

    2024 arXiv

  28. [36]

    In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar)

    Yin, H., Kononova, A.V., Bäck, T., van Stein, N.: Controlling the mutation in large language models for the efficient evolution of algorithms. In: International Conference on the Applications of Evolutionary Computation (Part of EvoStar). pp. 403–417. Springer (2025)

    2025

  29. [37]

    In: International Conference on Parallel Problem Solving from Nature

    Zhang, R., Liu, F., Lin, X., Wang, Z., Lu, Z., Zhang, Q.: Understanding the im- portance of evolutionary search in automated heuristic design with large language models. In: International Conference on Parallel Problem Solving from Nature. pp. 185–202. Springer (2024)

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.