Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Evolving proxy evaluators cut real evaluations by 59.1%

desk verdict A genuinely new co-evolution framework with plausible headline savings, but the load-bearing embedding assumption and thin statistics make the result conditional. read the letter →

arxiv 2608.08189 v1 pith:SOXON6SV submitted 2026-08-08 cs.AI

Ximeng Liu , Qianlong Wang , Yingming Mao , Annan Li , Yatao Li , Shizhen Zhao , Jianmin Wu , Dawei Yin
show 1 more author
Dou Shen
This is my paper · ORCID
classification cs.AI
keywords LLM-drivenprogramdiscoveryco-evolutionproxyevaluatorssurrogatemodelsexpensiveevaluationbudgetsdistributionshiftsearchscientificdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Janus's central claim is that the evaluator, not just the target program, should be evolved when real evaluations are expensive. Instead of fitting a fixed surrogate to scarce search-biased labels, it asks an LLM to generate executable proxy evaluators from domain descriptions, calibrates their parameters on real outcomes, and keeps a portfolio of region-specific proxies whose credits are updated online. Proxy predictions are used only to prioritize which candidates get real evaluation; a candidate must pass real validation before it can enter the population or update the incumbent. Across five scientific and engineering design tasks, Janus reports a larger best-so-far improvement over the real-evaluation budget and higher final performance than a matched algorithm-only baseline, reaching 99% of the baseline's final improvement with 59.1% fewer real evaluations on average. If the claim holds, it would let LLM-driven discovery work where trust is scarce and each evaluation costs hours or days.

What carries the argument

The load-bearing mechanism is the evaluator-as-program representation combined with real-anchored promotion. An evaluator is a pair $(g_\phi, \theta_\phi(\mathcal{A}))$: a program structure generated by an LLM from the task description and an archive of real outcomes, plus fitted parameters calibrated on that archive. The framework clusters the current target-program population by cosine distance between code embeddings, assigns each new candidate to the nearest region, and scores it with the top-credit evaluators in that region under a promotion-aligned meta-objective that rewards recovering near-optimal archive programs in the predicted top-$k$. Online credit updates adjust each evaluator's regional weight from the real outcomes of the candidates it helped promote. A candidate's acquisition score combines predicted quality, predicted validity, portfolio uncertainty, and embedding-region novelty, and only the real evaluator can confirm or reject the candidate.

What would settle it

Take a set of candidates already scored by the real evaluator in one Janus run, cluster them by code-embedding cosine distance, and compare within-cluster variance in real utility to between-cluster variance. If within-cluster variance is comparable to between-cluster variance, then the region-conditioned portfolio and novelty term are assigning meaning to code distances that do not correspond to real behavioral differences.

Watch

Extended reading notes

Core claim

The paper's central claim is that co-evolving evaluators with target programs transfers the benefits of evaluator-guided LLM search to settings with expensive real evaluation. The key move is to represent each proxy evaluator as an LLM-generated executable program with structure and fitted parameters, to score evaluators by how well their top-ranked candidates recover the best archive programs (a promotion-aligned objective), and to maintain region-conditioned portfolios with online credit updates as real outcomes arrive. Because proxies remain fallible, Janus never lets a proxy prediction promote a candidate alone: candidates are screened and ranked by the portfolio, but the real evaluator is the only authority that admits a program to the population or updates the incumbent. The evidence is five program-design domains with real evaluations ranging from full battery-cycle simulations to packet-level network simulations, where Janus reports higher area under the best-so-far improvement curve and higher terminal performance than the algorithm-only baseline, reaching the baseline's endpoint with 59.1% fewer real-evaluator calls on average. Evolved evaluators also rank promising candidates more accurately than their seed versions.

Load-bearing premise

The load-bearing assumption is that code similarity, measured as cosine distance between program embeddings, tracks the behavior the real evaluator cares about; Janus uses this distance to form regions, choose evaluators, and reward novelty, and the paper does not validate it.

Editorial extensions

If this is right

  • On the five test domains, Janus beats the algorithm-only baseline in both budget-averaged and terminal improvement; the gap in budget-averaged improvement ranges from 1.6 to 45.1 percentage points across domains.
  • Janus reaches 99% of the baseline's final improvement with 16.7%–82.8% fewer real-evaluator calls, with a mean reduction of 59.1%.
  • Evolved evaluators rank held-out candidates more accurately than their seed versions in the three reported domains, with NDCG@3 gains of 0.282, 0.694, and 0.311.
  • Ablations that remove evaluator evolution, the portfolio, the promotion-aligned objective, or online credit updates all reduce both metrics, indicating the four components address complementary failure modes.
  • In an objective-reweighting test, a previously evolved evaluator transferred without new labels and achieved an NDCG@3 of 0.959, showing the proxy's component structure is not tied to a single objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper's critique of its fixed-embedding Gaussian-process baseline also applies to its own region logic: cosine distance between code embeddings is never shown to track task-specific physical behavior. If similar code does not imply similar real-evaluator behavior, region-conditioned portfolios and novelty bonuses can misallocate the budget.
  • Inference: the reported savings are in real-evaluator calls; Janus spends 1.8–4.5 times as many LLM tokens per completed call as the algorithm-only baseline. The practical gain depends on real evaluations being so expensive that token cost is negligible, so on cheaper evaluation tasks the method could lose its edge.
  • Inference: a direct test of the framework's hypothesis would be to replace code-embedding similarity with behavior-based similarity from cheap low-fidelity simulations, which would separate the contribution of co-evolving evaluators from the contribution of the embedding similarity assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Janus, a framework that co-evolves LLM-generated target programs and LLM-generated executable proxy evaluators under a limited real-evaluation budget. Proxy evaluators are calibrated on a real-outcome archive, selected by a promotion-aligned meta-objective, organized into region-conditioned portfolios with online credit updates, and used only to prioritize candidates; real validation is required before any program can enter the target population or update the incumbent. Across five program-design benchmarks (battery fast charging, AQM, FDTD demultiplexing, reactor control, and perishable inventory routing), the authors report that Janus achieves larger area under the best-so-far improvement curve over the real-evaluation budget and higher final performance than a matched algorithm-only baseline, reaching 99% of the baseline's final improvement with on average 59.1% fewer real evaluations. The paper also reports improvements in evaluator ranking accuracy from evaluator evolution and a component ablation study.

Significance. If the result holds, Janus is a useful step toward making LLM-driven program discovery applicable when ground-truth evaluation is expensive. The method is principled in its real-anchored design: proxy predictions only rank candidates, promotion requires real validation, and the meta-objective directly targets the downstream search goal rather than generic regression accuracy. The ablations in Table III show that each component contributes something, and the evaluator-ranking gains in Figure 6 are concrete. The paper is also honest about token costs and includes a Direct-LLM replication protocol. The headline claim is not circular, because the central metric is anchored to real-evaluator outcomes and candidates must pass real validation before affecting the population. The main risks are empirical and methodological: an unvalidated similarity assumption that underpins the proposed mechanism, a mismatch between the motivating expensive evaluations and the actual simulated oracles, and insufficient statistical support for the headline numbers.

major comments (4)
  1. [§IV-B, Eq. (4), Eq. (6), and Alternative proxy baselines] The proposed mechanism—region-conditioned portfolios and the novelty term η(x)—rests on the assumption that cosine distance between code embeddings tracks behavior under the real evaluator, but the paper never specifies the embedding model, reports no correlation between embedding distance and real-evaluator agreement, and performs no ablation of the similarity measure. This is exactly the assumption the authors reject for Embed-GP when they write that 'semantic similarity may not preserve the task-specific numerical or physical behaviors governing performance.' Because the ablations in Table III show that evaluator evolution, portfolio selection, and online credits are all load-bearing, and because these components operate through the same embedding-defined regions, the reported gains cannot yet be attributed to the claimed mechanism. Please report the embedding model, provide a correlation analysis between dcos and real-evaluator score differences on held-out candidate pairs, and include an ablation that replaces cosine distance with an alternative (e.g., behavioral features or random region assignments).
  2. [§IV-A and Table II vs. Table I] The motivating examples are evaluations that take hours to days (Table I), but the five 'real' evaluators used in the experiments are lightweight simulations: 2D FDTD on a 480×320 grid, six 8-second ns-3 runs, 30-cycle PyBaMM at three temperatures, a stiff ODE, and a 36-episode IRP. The abstract's claim that Janus 'extends evaluator-guided LLM discovery... to scientific domains where trustworthy evaluation is scarce and expensive' is therefore not directly demonstrated; the observed 59.1% call reduction is measured under cheap, reproducible oracles. Either add at least one genuinely expensive evaluator (or a realistic cost model) to the benchmark suite, or explicitly restrict the claim to moderate-cost simulated oracles and discuss the transfer of the results to truly expensive settings.
  3. [§IV-A (Metrics) and §IV-B (Main Results)] Every method-domain condition is run with three random seeds and no significance tests or confidence intervals are reported. The headline mean reduction of 59.1% is computed across five domains with no uncertainty, and Table III reports overlapping standard deviations for several ablations (e.g., AQM rows S and A1). With n=3, the claims that Janus 'achieves' larger AUBC∆ in all five domains and that every ablated component is necessary are not strongly supported. Please provide per-seed results, paired tests or bootstrap intervals for the AUBC∆ and Calls@99% comparisons, and a statement of the variance of the 59.1% estimate across seeds and domains.
  4. [§III-A, Algorithm 1, Eq. (3), Eq. (5), Eq. (6)] The values of the free parameters λ_R, λ_N, λ_ρ, κ, w_q, w_u, w_n, τ_p, τ_e, b, K, and the archive bounds are not reported in the main text or appendices. These parameters control the balance among exploitation, uncertainty exploration, and novelty, as well as the periods at which evaluators are evolved and refreshed; without them the method is not reproducible and the sensitivity of the headline result to these choices is unknown. Report a full hyperparameter table and a sensitivity analysis for at least the acquisition weights (w_q, w_u, w_n) and the evolution periods (τ_p, τ_e).
minor comments (6)
  1. [Abstract] The abstract contains a typo: '99/% of the baseline's final improvement' should read '99% of the baseline's final improvement'.
  2. [§III-C, Eq. (5)] The factor α_{e,r} appears in Eq. (5) before it is defined; move the definition immediately before the equation to improve readability.
  3. [Figure 5] The shading spans only the minimum and maximum of three seeds, which can suggest more stability than present; plotting the three individual runs or adding standard-error bands would be more informative.
  4. [§IV-C, Figure 6] The 'highest-weight evolved evaluator' is chosen post hoc from the evolved evaluator population; please describe how this selection is made (e.g., by Fmeta on the archive) and clarify that the 40 later candidates are temporally separated from that selection, since this is the basis for the reported NDCG@3 gains.
  5. [§III-B, Eq. (2)] The paper never states which embedding model produces z(x); this information should be added to the reproducibility details, as the entire region-conditioning scheme depends on it.
  6. [§II-C] The sentence reporting a 'mean knee-location error of 0.500 tested sender-count positions' is confusing; clarify that this is 0.500 grid steps rather than a count of tested positions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Janus's headline gains are measured against real evaluator outcomes; proxies only prioritize candidates, and real validation gates the population, so the central claims do not reduce to fitted inputs or self-citations.

full rationale

The headline claim is anchored to the real evaluator: Eq. (1) optimizes observed real utility from H_B, Algorithm 1 lines 5-7 require a real evaluation before P_alg or the incumbent updates, and Section III-D states 'only a real-valid outcome may enter P_alg or update the incumbent.' The Calls@99% metric counts real-evaluator calls to reach 99% of Alg.-Only's terminal real improvement, so the efficiency claim is defined on real outcomes, not proxy scores. Evaluator fitting and selection use the real-outcome archive A via Eq. (3), but this is online calibration and selection, not a fitted parameter renamed as a prediction; the final metric is independent of the proxy. The evaluator-ranking gains in Section IV-C refit seed and evolved evaluators on the same checkpoint archive and test on 40 later candidates excluded from fitting, so the comparison is not a restatement of training loss. A mild self-referential risk exists because A is produced by the search the evaluator guides, and the paper itself acknowledges that 'real evaluations are limited and selected by the search process, making the available data both sparse and biased' (Section I), but it mitigates this with a balanced, deduplicated archive and with real-anchored promotion; this is a robustness and overfitting concern, not a definitional circularity. The cosine-embedding assumption in Eqs. (4) and (6) is unvalidated and is the same assumption the paper uses to criticize Embed-GP ('semantic similarity may not preserve the task-specific numerical or physical behaviors governing performance'), but this is an allocational assumption and a correctness risk, not a reduction of the central claim to its own inputs. Reference [40] is a same-group citation in related work and is not load-bearing. No circular step meets the evidentiary bar of Eq. X = Eq. Y by construction or fitted-input-renamed-as-prediction, so the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on several undocumented hyperparameters and on an unvalidated embedding-similarity assumption. No new physical or mathematical entities are introduced; all proxies are executable programs whose parameters are fitted to real outcomes.

free parameters (4)
  • lambda_R, lambda_N, lambda_rho
    Weights in the promotion-aligned meta-objective (Eq. 3). No values or tuning procedure are reported; they control how evaluators are selected.
  • kappa
    Online credit update hyperparameter in alpha_{e,r}=n/(n+kappa) (Eq. 5). No value given.
  • w_q, w_u, w_n
    Acquisition weights in Eq. (6). No values given; they balance exploitation, uncertainty, and novelty.
  • tau_p, tau_e, b, K, archive bounds
    Scheduling and portfolio hyperparameters in Algorithm 1: promotion period, evaluator-evolution period, refresh batch size, portfolio size, archive size. Only symbols are given.
assumptions (4)
  • domain assumption LLM domain knowledge encodes plausible proxy structures for each task.
    Section III.B relies on the LLM generating executable evaluator programs with 'plausible features, equations, and conditional structures' from the task description and archive diagnostics.
  • ad hoc to paper Cosine distance between code embeddings reflects real-evaluator behavior similarity.
    Used in Eq. (4) to define regions and in Eq. (6) for novelty. No empirical validation is provided, and the paper uses the same assumption to criticize Embed-GP.
  • domain assumption The real evaluator is the ground truth and is always available for validation.
    Underlies the problem statement in Eq. (1) and the real-anchored promotion rule in Section III.D. This is the standard oracle assumption in expensive-evaluation optimization.
  • standard math The AlphaEvolve island-based population search is a sound base for LLM program discovery.
    Janus builds on this framework (Section III.A) and draws on its diversity maintenance and parent selection; the base method is cited but not re-derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets." pith.science (2026). https://pith.science/paper/SOXON6SV

@misc{pith2026260808189,
  author       = {Pith},
  title        = {Pith review of: Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SOXON6SV}},
  note         = {Machine review of arXiv:2608.08189}
}
read the original abstract

LLM-driven program discovery relies on rapid evaluator feedback, but many scientific and engineering tasks require high-fidelity simulations, hardware execution, or physical experiments, making each evaluation expensive. Cheap surrogate evaluators can reduce this cost, yet fixed surrogates are vulnerable to search-induced distribution shift and are difficult to fit reliably from sparse, search-biased labels. We introduce Janus, a framework that uses LLMs to co-evolve target programs and executable proxy evaluators. To address label scarcity, Janus leverages domain knowledge encoded in LLMs to generate task-specific evaluator programs and calibrates them using real outcomes. To mitigate distribution shift, Janus evolves evaluators alongside target programs, selects them using a promotion-aligned objective, and maintains region-conditioned portfolios with online credit updates. Because proxy predictions remain fallible, Janus uses them only to prioritize candidates and requires real validation before candidates can enter the target-program population or update the incumbent. Across five scientific and engineering design tasks, Janus achieves a larger area under the best-so-far improvement curve over the real-evaluation budget and higher final performance than a matched baseline that evolves only target programs. On average, Janus reaches 99/% of the baseline's final improvement with 59.1/% fewer real evaluations. Evolved proxy evaluators also rank promising candidates more accurately than their seed versions. Together, these results extend evaluator-guided LLM discovery from tasks with cheap, scalable feedback to scientific domains where trustworthy evaluation is scarce and expensive.

Figures

Figures reproduced from arXiv: 2608.08189 by the authors.

Figure 1
Figure 1. AlphaEvolve-style evaluator-guided discovery loop. The LLM proposes executable [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Fixed-surrogate exploitation in GFP design. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Few-shot TCP-incast knee recovery. Together, the insights from § II-B and § II-C motivate Janus. To extend evaluator-guided discovery to scientific domains where real evaluation is expensive, Janus uses an LLM to generate and update proxy evaluators online from the task description and accumulated real labels. Janus therefore co￾evolves not only target programs but also the proxy evaluators that guide the search. II… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Results across five program-design domains under matched real-evaluation budgets. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Ranking-accuracy gains from evaluator evolution. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 41 canonical work pages

  1. [1]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=Bb4VGOWELI

    work page 2024

  2. [2]

    ReEvo: Large language models as hyper-heuristics with reflective evolution,

    H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, “ReEvo: Large language models as hyper-heuristics with reflective evolution,” inAdvances in Neural Information Processing Systems, vol. 37, 2024. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 4ced59d480e07d290b6f29fc8798f195-Abstract-Conference.html

    work page 2024

  3. [3]

    Symbolic regression with a learned concept library,

    A. Grayeli, A. Sehgal, O. Costilla-Reyes, M. Cranmer, and S. Chaudhuri, “Symbolic regression with a learned concept library,” inAdvances in Neural Information Processing Systems, vol. 37, 2024. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 4ec3ddc465c6d650c9c419fb91f1c00a-Abstract-Conference.html

    work page 2024

  4. [4]

    Synthesizing programmatic reinforcement learning policies with large language model guided search,

    M. Liu, C.-H. Yu, W.-H. Lee, C.-W. Hung, Y .-C. Chen, and S.-H. Sun, “Synthesizing programmatic reinforcement learning policies with large language model guided search,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://proceedings.iclr.cc/paper_files/paper/2025/ hash/dcf887f2bfe2776584e3bce80ed206ef...

    work page 2025

  5. [5]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,”Nature, vol. 625, pp. 468–475, 2024

    work page 2024

  6. [6]

    AlphaEvolve: A coding agent for scientific and algorithmic discovery,

    A. Novikov, N. V ˜u, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog, “AlphaEvolve: A coding agent for scientific and algorithmic discovery,” 2025. [Online]. Available: https://arxiv.org/abs/2506.13131

    arXiv 2025

  7. [7]

    Language modeling by language models,

    J. Cheng, P. Clark, and K. Richardson, “Language modeling by language models,” inAdvances in Neural Information Processing Systems, vol. 38, 2025. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2025/hash/ fa40454addc83f1f63ddebe65aa8fb39-Abstract-Conference.html

    work page 2025

  8. [8]

    BoTorch: A framework for efficient monte-carlo bayesian optimization,

    M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy, “BoTorch: A framework for efficient monte-carlo bayesian optimization,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 21 524–21 538. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/ f5b1b89d98b7286673128a5fb112cb9a-Abstract.html

    work page 2020

Show all 45 references
  1. [9]

    Scalable global optimization via local Bayesian optimization,

    D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek, “Scalable global optimization via local Bayesian optimization,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2019/h...

    2019

  2. [10]

    Conditioning by adaptive sampling for robust design,

    D. Brookes, H. Park, and J. Listgarten, “Conditioning by adaptive sampling for robust design,” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 773–782. [Online]. Available: https://...

    2019

  3. [11]

    Conservative objective models for effective offline model-based optimization,

    B. Trabucco, A. Kumar, X. Geng, and S. Levine, “Conservative objective models for effective offline model-based optimization,” inProceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol

  4. [12]

    Generative adversarial model-based optimization via source critic regularization,

    M. S. Yao, Y . Zeng, H. Bastani, J. Gardner, J. C. Gee, and O. Bastani, “Generative adversarial model-based optimization via source critic regularization,” inAdvances in Neural Information Processing Systems, vol. 37, 2024. [Online]. Available: https://proceedings.neurips.cc/p...

    2024

  5. [13]

    Fast charging of Li-ion batteries via learning and optimization,

    R. Rodriguez, Y . Wang, and D. Soudbakhsh, “Fast charging of Li-ion batteries via learning and optimization,” in2024 European Control Conference (ECC), 2024, pp. 3715–3720

    2024

  6. [14]

    DESiRED: Dynamic, enhanced, and smart iRED: A P4-AQM with deep reinforcement learning and in-band network telemetry,

    L. C. de Almeida, W. R. Dias da Silva, T. C. Tavares, R. Pasquini, C. Papagianni, and F. L. Verdi, “DESiRED: Dynamic, enhanced, and smart iRED: A P4-AQM with deep reinforcement learning and in-band network telemetry,”Computer Networks, vol. 244, p. 110326, 2024

    2024

  7. [15]

    Inverse-designed low-crosstalk CWDM (de)multiplexer assisted by photonic crystals,

    R. Wu, F. Ding, F. Li, and Y . Liu, “Inverse-designed low-crosstalk CWDM (de)multiplexer assisted by photonic crystals,”Journal of Lightwave Technology, vol. 42, no. 14, pp. 4899–4905, 2024

    2024

  8. [16]

    A hierarchical multi-parametric programming approach for dynamic risk-based model predictive quality control,

    A. Braniff and Y . Tian, “A hierarchical multi-parametric programming approach for dynamic risk-based model predictive quality control,” Control Engineering Practice, vol. 152, p. 106062, 2024

    2024

  9. [17]

    Inventory management and delivery of perishable products with stochastic demands and risks consideration,

    A. Violi, A. De Maio, and G. Fattoruso, “Inventory management and delivery of perishable products with stochastic demands and risks consideration,”Procedia Computer Science, vol. 232, pp. 2941–2949, 2024

    2024

  10. [18]

    Data-driven prediction of battery cycle life before capacity degradation,

    K. A. Severson, P. M. Attia, N. Jin, N. Perkins, B. Jiang, Z. Yang, M. H. Chen, M. Aykol, P. K. Herring, D. Fraggedakis, M. Z. Bazant, S. J. Harris, W. C. Chueh, and R. D. Braatz, “Data-driven prediction of battery cycle life before capacity degradation,”Nature Energy, vol. 4,...

    2019

  11. [19]

    Closed-loop optimization of fast-charging protocols for batteries with machine learning,

    P. M. Attia, A. Grover, N. Jin, K. A. Severson, T. M. Markov, Y .-H. Liao, M. H. Chen, B. Cheong, N. Perkins, Z. Yang, P. K. Herring, M. Aykol, S. J. Harris, R. D. Braatz, S. Ermon, and W. C. Chueh, “Closed-loop optimization of fast-charging protocols for batteries with machin...

    2020

  12. [20]

    Scalable tail latency estimation for data center networks,

    K. Zhao, P. Goyal, M. Alizadeh, and T. E. Anderson, “Scalable tail latency estimation for data center networks,” in20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, 2023, pp. 685–702. [Online]. Available: https://www.usenix.org...

    2023

  13. [21]

    Automated optimization of photonic crystal slab cavities,

    M. Minkov and V . Savona, “Automated optimization of photonic crystal slab cavities,”Scientific Reports, vol. 4, no. 1, p. 5124, 2014

    2014

  14. [22]

    Gaussian elimination is not optimal,

    V . Strassen, “Gaussian elimination is not optimal,”Numerische Mathe- matik, vol. 13, pp. 354–356, 1969

    1969

  15. [23]

    Network simulations with the ns-3 simulator,

    T. R. Henderson, M. Lacage, G. F. Riley, C. Dowell, and J. Kopena, “Network simulations with the ns-3 simulator,” inProceedings of the ACM SIGCOMM 2008 Conference on Data Communication, 2008, pp. 527–528

    2008

  16. [24]

    Practical bayesian optimization of machine learning algorithms,

    J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” inAdvances in Neural Information Processing Systems, vol. 25, 2012, pp. 2951–2959. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2012/hash/05311...

    2012

  17. [25]

    Local fitness landscape of the green fluorescent protein,

    K. S. Sarkisyan, D. A. Bolotin, M. V . Meer, D. R. Usmanova, A. S. Mishin, G. V . Sharonov, D. N. Ivankov, N. G. Bozhanova, M. S. Baranov, O. Soylemez, N. S. Bogatyreva, P. K. Vlasov, E. S. Egorov, M. D. Logacheva, A. S. Kondrashov, D. M. Chudakov, E. V . Putintseva, I. Z. Mam...

    2016

  18. [26]

    Evolutionary model type selection for global surrogate modeling,

    D. Gorissen, T. Dhaene, and F. De Turck, “Evolutionary model type selection for global surrogate modeling,”Journal of Machine Learning Research, vol. 10, pp. 2039–2078, 2009. [Online]. Available: https://www.jmlr.org/papers/v10/gorissen09a.html

    2009

  19. [27]

    Large language models as surrogate models in evolutionary algorithms: A preliminary study,

    H. Hao, X. Zhang, and A. Zhou, “Large language models as surrogate models in evolutionary algorithms: A preliminary study,”Swarm and Evolutionary Computation, vol. 91, p. 101741, 2024

    2024

  20. [28]

    Random early detection gateways for congestion avoidance,

    S. Floyd and V . Jacobson, “Random early detection gateways for congestion avoidance,”IEEE/ACM Transactions on Networking, vol. 1, no. 4, pp. 397–413, 1993

    1993

  21. [29]

    Agri-food supply chains with stochastic demands: A multi-period inventory routing problem with perishable products,

    B. S. Onggo, J. Panadero, C. G. Corlu, and A. A. Juan, “Agri-food supply chains with stochastic demands: A multi-period inventory routing problem with perishable products,”Simulation Modelling Practice and Theory, vol. 97, p. 101970, 2019

    2019

  22. [30]

    Evolution of heuristics: Towards efficient automatic algorithm design using large language model,

    F. Liu, T. Xialiang, M. Yuan, X. Lin, F. Luo, Z. Wang, Z. Lu, and Q. Zhang, “Evolution of heuristics: Towards efficient automatic algorithm design using large language model,” inProceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine L...

    2024

  23. [31]

    Monte carlo tree search for comprehensive exploration in LLM-based automatic heuristic design,

    Z. Zheng, Z. Xie, Z. Wang, and B. Hooi, “Monte carlo tree search for comprehensive exploration in LLM-based automatic heuristic design,” inProceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267, 2025, pp. 7...

    2025

  24. [32]

    Multi-fidelity Bayesian optimisation with continuous approximations,

    K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póczos, “Multi-fidelity Bayesian optimisation with continuous approximations,” inProceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70, 2017, pp. 1799–1808. ...

    2017

  25. [33]

    Surrogate model assisted cooperative coevolution for large scale opti- mization,

    Z. Ren, B. Pang, M. Wang, Z. Feng, Y . Liang, A. Chen, and Y . Zhang, “Surrogate model assisted cooperative coevolution for large scale opti- mization,”Applied Intelligence, vol. 49, no. 2, pp. 513–531, 2019

    2019

  26. [34]

    EZNAS: Evolving zero-cost proxies for neural architecture scoring,

    Y . Akhauri, J. P. Muñoz, N. Jain, and R. Iyer, “EZNAS: Evolving zero-cost proxies for neural architecture scoring,” inAdvances in Neural Information Processing Systems, vol. 35, 2022. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/hash/ c474dc287c6...

    2022

  27. [35]

    Revolutionizing training-free NAS: Towards efficient automatic proxy discovery via large language models,

    H. Kang, L. Lin, and H. Wang, “Revolutionizing training-free NAS: Towards efficient automatic proxy discovery via large language models,” inAdvances in Neural Information Processing Systems, vol. 38, 2025. [Online]. Available: https://openreview.net/forum?id=3naHyE5klE

    2025

  28. [36]

    Co- evolution of large language models and configuration strategies to enhance surrogate-assisted evolutionary algorithm,

    L. Xie, Y . Zhang, Z. Tang, E. Chung, G. Li, and Z. Wang, “Co- evolution of large language models and configuration strategies to enhance surrogate-assisted evolutionary algorithm,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025, p...

    2025

  29. [37]

    Large language model-driven surrogate-assisted evolutionary algorithm for expensive optimization,

    L. Xie, G. Li, Z. Wang, E. Chung, and M. Gong, “Large language model-driven surrogate-assisted evolutionary algorithm for expensive optimization,” 2025. [Online]. Available: https://arxiv.org/abs/2507.02892

    2025 arXiv

  30. [38]

    Eureka: Human-level reward design via coding large language models,

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y . Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https:/...

    2024

  31. [39]

    LaRes: Evolutionary reinforcement learning with LLM-based adaptive reward search,

    P. Li, H. Tang, J. Qiao, Y . Zheng, and J. Hao, “LaRes: Evolutionary reinforcement learning with LLM-based adaptive reward search,” in Advances in Neural Information Processing Systems, vol. 38, 2025. [Online]. Available: https://openreview.net/forum?id=jRjvcqtdtA

    2025

  32. [40]

    Beyond static evaluation: Co- evolutionary mechanisms for LLM-driven strategy evolution in adversarial games,

    H. Li, Z. Ge, Z. Zhang, X. Yuan, Y . Lo, Q. Liu, B. An, D. Rong, J. Liu, A. Li, J. Wu, D. Yin, and D. Shen, “Beyond static evaluation: Co- evolutionary mechanisms for LLM-driven strategy evolution in adversarial games,” 2026. [Online]. Available: https://arxiv.org/abs/2606.10389

    2026 arXiv

  33. [41]

    Game-theoretic co-evolution for LLM-based heuristic discovery,

    X. Ke, K. Li, J. Xing, Y . Zhang, and J. Cheng, “Game-theoretic co-evolution for LLM-based heuristic discovery,” 2026. [Online]. Available: https://arxiv.org/abs/2601.22896

    2026

  34. [42]

    Self-evolving deep research via joint generation and evaluation,

    H. Zhu, C. Cai, Y . Song, X. Chen, S. Han, and Y . Guo, “Self-evolving deep research via joint generation and evaluation,” 2026. [Online]. Available: https://arxiv.org/abs/2606.04507

    2026 arXiv

  35. [43]

    Python Battery Mathematical Modelling (PyBaMM),

    V . Sulzer, S. G. Marquis, R. Timms, M. Robinson, and S. J. Chapman, “Python Battery Mathematical Modelling (PyBaMM),”Journal of Open Research Software, vol. 9, no. 1, p. 14, 2021

    2021

  36. [44]

    Machine learning approaches for active queue management: A survey, taxonomy, and future directions,

    M. P. Toopchinezhad and M. Ahmadi, “Machine learning approaches for active queue management: A survey, taxonomy, and future directions,” Computer Networks, vol. 262, p. 111174, 2025. APPENDIXA BENCHMARKDETAILS Each benchmark asks the search procedure to produce an executable p...

    2025

  37. [139]

    10 358–10 368

    PMLR, 2021, pp. 10 358–10 368. [Online]. Available: https: //proceedings.mlr.press/v139/trabucco21a.html

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.