Pith. sign in

REVIEW 2 major objections 6 minor 61 references

Better Together, in the Right Order: Classical-then-LLM Optimization for SE

T0 review · 2 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For SE configuration, a cheap classical learner should seed the LLM, not the reverse: SNAP2 tops 85% of 105 tasks while using fewer tokens.

desk verdict Classical-then-LLM (SNAP2) beats LLM-first hybrids and pure SNAP on 105 MOOT tasks under a shared snap-to-table protocol; the ordering result is solid within that regime, and the projection limit is already the right caveat. read the letter →

arxiv 2607.02583 v1 pith:K3GIKUET submitted 2026-07-01 cs.SE

Srinath Srinivasan , Tim Menzies This is my paper
classification cs.SE
keywords softwareengineeringoptimizationlargelanguagemodelswarm-startactivelearningmulti-objectiveconfigurationtuningclassical-then-LLMSNAP2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual way of mixing large language models with classical optimizers for software-engineering configuration is backwards. Most prior hybrids let a classical search loop own the process and call the LLM only for warm starts, mutations, or surrogates. The authors reverse that order: a cheap classical active learner (EZR) spends the first half of a tight labeling budget to gather informative rows, then an OPRO-style LLM finishes the remaining labels. Their method, SNAP2, reaches the top statistical tier on 85% of 105 real multi-objective SE tasks, beating the same LLM run alone (75%) and every classical-led hybrid, while using roughly 30% fewer tokens and running 1.4 times faster than pure LLM search. The practical claim is simple: do not study classical learners or LLMs in isolation, and when you combine them, let the inexpensive classical step do the early work so the LLM only has to close.

What carries the argument

SNAP2: replace the random warm-start of an OPRO-style LLM re-prompting loop with the labeled trajectory produced by a short EZR run (first half of a fixed B=20 labeling budget), then let the LLM finish the rest under the same total budget and collision-tracking diversification.

What would settle it

On the same 105 tasks and budget, a pure generative LLM that is free to propose and evaluate configurations never previously measured (no nearest-row projection) would reverse the ranking so that classical-first SNAP2 no longer dominates pure LLM or LLM-first hybrids on top-tier rate and cost.

Watch

Extended reading notes

Core claim

Across 105 MOOT software-engineering optimization tasks, the hybrid that first runs a short classical active learner (EZR for 10 labels) and then hands its trajectory to an LLM for the remaining 10 labels (SNAP2) is the single best of seven methods spanning pure classical, pure LLM, and LLM-first hybrids. It reaches the top tier on 85% of tasks, above the same LLM alone (75%) and above every method in which the classical optimizer keeps control, while also cutting tokens by about 30% and wall time by a factor of 1.4 relative to the pure LLM baseline.

Load-bearing premise

Every LLM proposal is snapped to the nearest already-measured row in a fixed table of configurations; if good answers lie off that table or the table is sparse, the ranking and the classical-then-LLM advantage need not hold.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies how classical optimizers and LLMs should be combined for SE configuration tasks on the MOOT corpus. It distinguishes three regimes (LLM-only opt1, classical-in-charge with LLM assist opt2, classical-only opt3) and introduces SNAP2, which reverses the usual warm-start order: a short EZR run (B=10) seeds an OPRO-style LLM loop that finishes the remaining labels under a fixed total budget B=20. On 105 tabular multi-objective SE tasks with 20 seeds each, SNAP2 reaches the top tier on 85% of tasks, ahead of pure SNAP (75%), the LLM-first opt2 methods (BS_LLM 74%, SYNTHCORE 70%), and EZR alone (71%), while using roughly 30% fewer tokens and running 1.4x faster than pure SNAP. The authors conclude that combination beats either alone and that classical-then-LLM order matters.

Significance. If the result holds under the stated regime, it is a clear and actionable contribution for SE configuration tuning: the field has largely studied LLM-first hybrids, and this paper shows that reversing the order yields both higher top-tier rate and lower cost on a large, real-task corpus. Strengths include the scale (105 datasets, 20 seeds), dual distribution-free tests for top-tier membership, explicit cost tables, open release of code/prompts/data, and an honest threats section that already bounds shared-machinery and projection concerns. The practical guidance (try free EZR first; escalate to classical-seeded LLM for mission-critical work) is useful. The result is scoped to tabular selection with nearest-row projection; that limit is load-bearing but already flagged by the authors.

major comments (2)
  1. External validity of the projection step (Algorithm 2 lines 11–15; §VI-A.e): every LLM proposal is snapped to the nearest already-measured row by feature distance ρ. All opt1/opt2 rankings, the 85% top-tier claim, and the cost advantage over pure SNAP are measured under this protocol. The paper correctly flags that this bounds the claim to choosing among pre-scored configurations and does not cover generative SE tasks (test generation, patch synthesis, off-table optima). For the central claim as stated this is acceptable, but the abstract and conclusion should state the tabular-selection scope more prominently so readers do not over-generalize the ordering result.
  2. Shared-substrate risk for SNAP2 vs EZR (§VI-A.c; Table II; Algorithm 2 WARMSTART v=2): SNAP2 reuses EZR as its first stage and is then ranked against standalone EZR. The authors’ mitigations are real—the EZR stage receives only B=10 (weaker than the B=20 baseline) yet SNAP2 still wins, and SNAP2 also beats LLM-first methods that share SNAP’s machinery—so the ordering claim is not circular. Still, the paper would be stronger with one additional control that seeds SNAP from a non-EZR classical method (e.g. RRP or random rare-pair) under the same B=10 split, to show the gain is not EZR-specific.
minor comments (6)
  1. Abstract and §I: “over 100 SE tasks” / “105” is fine, but the common-denominator filter (only tasks where all seven methods complete) should be mentioned once in the abstract or early intro so the denominator is transparent.
  2. Table IV and §V RQ1: the 70–75% cluster is correctly described as statistically hard to separate; a short note that top-tier rate can credit multiple methods per dataset would help readers interpret the percentages.
  3. Fig. 3 caption: clarify that two of 105 datasets lack a defined Δ baseline and are omitted from the plot.
  4. §IV-E / Table III: the B=20 budget argument is useful but rests on idealized assumptions the authors already qualify; a one-sentence pointer that empirical need was B=20 (not the theoretical <8) would tighten the presentation.
  5. Notation: d2h is defined in Eq. (1) and used throughout; ensure the first use in the algorithms section points back to that equation for readers who skip §IV-B.
  6. References: Ganguly et al. FSE’26 is cited as the classical SOTA anchor; if that work is still under review or forthcoming, note the status so the dependency is clear.

Circularity Check

1 steps flagged · score 1.0 of 10

No derivation-by-construction circularity; only a minor, author-flagged shared-substrate risk that does not force the ranking.

  1. other [§III-C / Alg. 2 (WARMSTART v=2) and §VI-A.c Shared Machinery]
    "SNAP2 replaces SNAP’s random warm start with the output of a short EZR run. EZR spends the first half of its labeling budget acquiring informative rows. … SNAP2 uses EZR as its first stage, then is compared against EZR in the same ranking. The two are not independent, so SNAP2’s advantage could reflect that shared substrate rather than the classical-then-LLM order. Three points bound this. … SNAP2 gives its EZR stage only B=10 labels, half the B=20 that the standalone EZR baseline gets … yet SNAP2 still wins."

    SNAP2 re-uses the EZR active-learning loop as its seed and is then ranked against standalone EZR. This creates a superficial shared-substrate appearance. It is not circular by construction: the seed is deliberately weaker (B=10 vs B=20), the split was fixed a priori, and SNAP2 also outperforms the LLM-first methods that share SNAP’s later machinery but reverse the order. The paper itself flags and bounds the risk; the ranking is not forced by the reuse.

full rationale

This is an empirical bake-off of seven optimizers (opt1/opt2/opt3) on 105 MOOT tabular SE tasks under a fixed label budget B=20 and a common nearest-row projection. The central claim (SNAP2 top-tier on 85% of tasks, ahead of SNAP alone and of every classical-controlled method) is a measured ranking under explicit statistical tests (Cliff’s δ + KS), not a first-principles derivation that reduces to its inputs. SNAP2 is defined as EZR(B=10) warm-start followed by the OPRO-style SNAP loop; the authors themselves surface the shared-substrate concern in §VI-A.c and bound it with three independent checks (fixed split, weaker B=10 stage still lifts the hybrid, and SNAP2 also beats LLM-first methods that share SNAP’s machinery but reverse order). Those checks prevent the ranking from being true by construction. Self-citations (Ganguly et al. for EZR’s classical strength, Senthilkumar et al. for the opt2 baselines) supply prior empirical baselines, not uniqueness theorems or load-bearing ansätze that forbid alternatives. The nearest-row projection (Alg. 2) is a methodological limit already listed under external validity, not a circular step. No fitted parameter is renamed a prediction, no equation is definitionally equivalent to its target, and no uniqueness result is imported from the same authors to force the ordering claim. Score 1 only for the residual shared-substrate appearance that the paper already mitigates; the derivation chain itself is self-contained against the stated evaluation regime.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a fixed labeling budget, a table-projection evaluation model, a specific classical active learner (EZR), one LLM, and statistical top-tier aggregation. No new physical entities are postulated. Free parameters are the usual experimental knobs (B, W, K, model, thresholds) chosen for cost and fairness rather than fitted to maximize the headline gap after the fact, though SNAP2 itself was noticed post-hoc.

free parameters (5)
  • labeling budget B = 20
    Fixed at 20 for all methods (SNAP2 splits 10+10). Justified by cost envelope and a Hamlet/Cohen argument (Table III), not cross-validated per dataset.
  • SNAP2 warm-start split W / EZR stage budget = 10
    First half of B given to EZR (B=10). Fixed in advance per authors, but the hybrid itself was discovered after seeing results.
  • LLM (openai/gpt-oss-120b) and decoding hyperparameters = gpt-oss-120b, T=0.7
    Single model, temperature 0.7, top-p 1.0, context/output caps. Results may not transfer to other models.
  • top-tier statistical thresholds = δ=0.195, KS 95%
    Cliff's δ < 0.195 and KS critical value 1.36*sqrt((n+m)/nm) define indistinguishability. Standard but still chosen cutoffs that control the 85% headline rate.
  • candidates per LLM call K = 2
    K=2 in the SNAP/SNAP2 loop. Affects token use and exploration.
assumptions (5)
  • domain assumption Sequential model-based optimization with a small active-learning budget is the right meta-procedure for SE configuration (cheap to list options, expensive to measure).
    Stated in Section III and used to justify B=20 and the shared SMO loop for all seven methods.
  • domain assumption Projecting a proposed configuration onto the nearest measured table row by feature distance ρ yields a valid performance score for ranking optimizers.
    Core evaluation model for SNAP/SNAP2/opt2 (Algorithm 2); external-validity threat VI-A.e acknowledges this bounds claims to on-table selection.
  • domain assumption EZR is a strong classical baseline (beats TPE, SMAC3, DEHB on MOOT).
    Adopted from Ganguly et al. FSE'26 without re-running those classical methods here (Section II-B.3).
  • ad hoc to paper Top-tier rate under Cliff's δ + KS is the primary quality metric for comparing optimizers across heterogeneous multi-objective tasks.
    Defined in Section IV-D; aggregates 105 per-dataset significance tests into the 85%/75%/71% headline numbers.
  • domain assumption d2h (normalized distance to utopia) and Δ improvement are adequate scalarizations for multi-objective SE tasks.
    Section IV-B; authors note HV/Chebyshev rankings are consistent but prefer d2h for SE landscape reasons.
invented entities (2)
  • SNAP2 (classical-then-LLM hybrid)
    purpose: Name and operationalize the reverse-order combination that fills the empty cell of Fig. 1.
    New method name for EZR warm-start + OPRO-style LLM finish; not a physical entity, but the paper's central invented construct.
  • Collision tracking (coll set) in SNAP
    purpose: Diversify LLM proposals away from already-evaluated neighborhoods without an external searcher.
    Algorithmic addition to OPRO described in Algorithm 2 and Fig. 2; internal mechanism, no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Better Together, in the Right Order: Classical-then-LLM Optimization for SE." pith.science (2026). https://pith.science/paper/K3GIKUET

@misc{pith2026260702583,
  author       = {Pith},
  title        = {Pith review of: Better Together, in the Right Order: Classical-then-LLM Optimization for SE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3GIKUET}},
  note         = {Machine review of arXiv:2607.02583}
}
read the original abstract

A growing body of work combines large language models (LLMs) with classical optimizers for software engineering (SE) configuration tasks. Often, the classical optimizer is in charge: it owns the search loop and calls the LLM only to assist in subroutines (e.g. to warm-start the first generation, propose a mutation, or stand in as a surrogate). We report that there is much value in the reverse approach: seeding an LLM with the results from a cheap classical learner. We call this method SNAP2. Applied to over 100 SE tasks, it is the single best of all methods studied, reaching the top tier on 85% of tasks, ahead of the same LLM run alone (75%) and ahead of every method in which the classical optimizer retains control. It is also less expensive: relative to the LLM-alone method, it uses roughly 30% fewer tokens and runs 1.4x faster, since the classical setup performs the inexpensive work, and the LLM is invoked only to finish. We conclude that it is unwise to study classical learners or LLMs in isolation: there is much value in combining the two, and in the order that combination is applied.

Figures

Figures reproduced from arXiv: 2607.02583 by the authors.

Figure 1
Figure 1. Methods grouped by who controls the search. Left: a classical optimizer [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example SNAP prompt, on a 3-feature task. The eight blocks match SNAP’s prompt template: (1) System sets the role and the strict in-range rule; (2) Hard constraints list each feature’s legal range; (3) Objectives are described but never proposed (they are scored for the model); (4) Trajectory lists tried configurations sorted worst-to-best, with the best placed last so it sits in the high-attention end-of-prompt … view at source ↗
Figure 3
Figure 3. Per-dataset mean ∆ (higher is better, see Eq. (2)) across the 103 of 105 datasets with a defined ∆ baseline (2 have none), sorted left to right by SNAP2 performance. Black line is SNAP2. Top: SNAP2 against the weaker methods (SYNTHCORE, RANDOM, RRP); SNAP2 sits above them on most datasets. Bottom: SNAP2 against the strongest competitors (SNAP, BS_LLM, EZR) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 2 canonical work pages

  1. [1]

    Minimal data, maximum clarity: A heuristic for explaining optimization,

    A. Rayegan and T. Menzies, “Minimal data, maximum clarity: A heuristic for explaining optimization,”Journal of Systems and Software, p. 112897, 2026

    2026

  2. [2]

    Promisetune: Unveiling causally promising and explainable configuration tuning,

    P. Chen and T. Chen, “Promisetune: Unveiling causally promising and explainable configuration tuning,” in48th International Conference on Software Engineering. IEEE, 2026

    2026

  3. [3]

    Can large language models improve se active learning via warm-starts?

    L. Senthilkumar and T. Menzies, “Can large language models improve se active learning via warm-starts?”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  4. [4]

    Accuracy can lie: On the impact of surrogate model in configuration tuning,

    P. Chen, J. Gong, and T. Chen, “Accuracy can lie: On the impact of surrogate model in configuration tuning,”IEEE Transactions on Software Engineering, vol. 51, no. 2, pp. 548–580, 2025

    2025

  5. [5]

    Learning from very little data: On the value of landscape analysis for predicting software project health,

    A. Lustosa and T. Menzies, “Learning from very little data: On the value of landscape analysis for predicting software project health,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, pp. 1–22, 2024

    2024

  6. [6]

    iSNEAK: Partial ordering as heuristics for model-based reasoning in software engineering,

    ——, “iSNEAK: Partial ordering as heuristics for model-based reasoning in software engineering,”IEEE Access, vol. 12, pp. 142 915–142 929, 2024

    2024

  7. [7]

    Beyond evolutionary algorithms for search-based software engineering,

    J. Chen, V . Nair, and T. Menzies, “Beyond evolutionary algorithms for search-based software engineering,”Information and Software Technology, vol. 95, pp. 281–294, 2018

    2018

  8. [8]

    EA Sports FC 25 database — ratings and stats,

    nyagami, “EA Sports FC 25 database — ratings and stats,” 2025, kaggle dataset

    2025

Show all 61 references
  1. [9]

    Telco customer churn,

    blastchar, “Telco customer churn,” 2025, kaggle dataset

    2025

  2. [10]

    Life expectancy (WHO) dataset,

    K. A. Rajarshi, “Life expectancy (WHO) dataset,” 2025, kaggle dataset

    2025

  3. [11]

    Marketing analytics — marketing data,

    J. Daoud, “Marketing analytics — marketing data,” 2022, kaggle dataset

    2022

  4. [12]

    An empirical study on configuration errors in commercial and open source systems,

    Z. Yin, X. Ma, J. Zheng, Y . Zhou, L. N. Bairavasundaram, and S. Pasupathy, “An empirical study on configuration errors in commercial and open source systems,” inProceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, 2011, pp. 159–172

    2011

  5. [13]

    Fail through the cracks: Cross-system interaction failures in modern cloud systems,

    L. Tang, C. Bhandari, Y . Zhang, A. Karanika, S. Ji, I. Gupta, and T. Xu, “Fail through the cracks: Cross-system interaction failures in modern cloud systems,” inProceedings of the Eighteenth European Conference on Computer Systems, 2023, pp. 433–451

    2023

  6. [14]

    Uncertainty-aware self-adaptation in cloud computing,

    P. Jamshidi and G. Casale, “Uncertainty-aware self-adaptation in cloud computing,” Ph.D. dissertation, Imperial College London, 2016, source for the STORM480x degradation result

    2016

  7. [15]

    Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software,

    T. Xu, L. Jin, X. Fan, Y . Zhou, S. Pasupathy, and R. Talwadker, “Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ...

    2015 doi

  8. [16]

    Automatic database management system tuning through large-scale machine learning,

    D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang, “Automatic database management system tuning through large-scale machine learning,” in Proceedings of the 2017 ACM international conference on management of data, 2017, pp. 1009–1024

    2017

  9. [17]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,”IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2020

    2020

  10. [18]

    Efficient compiler autotuning via bayesian optimization,

    J. Chen, N. Xu, P. Chen, and H. Zhang, “Efficient compiler autotuning via bayesian optimization,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1198– 1209

    2021

  11. [19]

    Performance- influence models for highly configurable systems,

    N. Siegmund, A. Grebhahn, S. Apel, and C. Kästner, “Performance- influence models for highly configurable systems,” inProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 284–294

    2015

  12. [20]

    Sequential model- based optimization for general algorithm configuration,

    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model- based optimization for general algorithm configuration,” inInternational conference on learning and intelligent optimization. Springer, 2011, pp. 507–523

    2011

  13. [21]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 12 028–12 068

    2024

  14. [22]

    Large language models as evolutionary optimizers,

    S. Liu, C. Chen, X. Qu, K. Tang, and Y .-S. Ong, “Large language models as evolutionary optimizers,” in2024 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2024, pp. 1–8

    2024

  15. [23]

    Large language models to enhance bayesian optimization,

    T. Liu, N. Astorga, N. Seedat, and M. van der Schaar, “Large language models to enhance bayesian optimization,”The Twelfth International Conference on Learning Representations, 2024

    2024

  16. [24]

    Multi- objective hierarchical optimization with large language models,

    A. Schwanke, L. Ivanov, D. Salinas, F. Hutter, and A. Zela, “Multi- objective hierarchical optimization with large language models,”arXiv preprint arXiv:2601.13892, 2026

    2026

  17. [25]

    Leveraging large language models for dynamic multi-objective optimization in UA V sensor-target assignment,

    Y . Zhang, G. Yi, H. Geng, and H. Wang, “Leveraging large language models for dynamic multi-objective optimization in UA V sensor-target assignment,”IFAC PapersOnLine, vol. 59, no. 20, pp. 2314–2319, 2025

    2025

  18. [26]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,”Nature, vol. 625, no. 7995, pp. 468–475, 2024

    2024

  19. [27]

    Exploring the true potential: Evaluating the black-box optimization capability of large language models,

    B. Huang, X. Wu, Y . Zhou, J. Wu, L. Feng, R. Cheng, and K. C. Tan, “Exploring the true potential: Evaluating the black-box optimization capability of large language models,”arXiv preprint arXiv:2404.06290, 2024

    2024 arXiv

  20. [28]

    How low can you go? the data-light se challenge,

    K. K. Ganguly and T. Menzies, “How low can you go? the data-light se challenge,” inFSE’26, 2026

    2026

  21. [29]

    SMAC3: A versatile Bayesian optimization package for hyperparameter optimization,

    M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter, “SMAC3: A versatile Bayesian optimization package for hyperparameter optimization,”Journal of Machine Learning Research, vol. 23, no. 54, pp. 1–9, 2022

    2022

  22. [30]

    Algorithms for hyper- parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. Kégl, “Algorithms for hyper- parameter optimization,”Advances in neural information processing systems, vol. 24, 2011

    2011

  23. [31]

    Botorch: A framework for efficient monte-carlo bayesian optimization,

    M. Balandat, B. Karrer, D. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy, “Botorch: A framework for efficient monte-carlo bayesian optimization,”Advances in neural information processing systems, vol. 33, pp. 21 524–21 538, 2020

    2020

  24. [32]

    Moea/d: A multiobjective evolutionary algorithm based on decomposition,

    Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,”IEEE Transactions on evolutionary computa- tion, vol. 11, no. 6, pp. 712–731, 2007

    2007

  25. [33]

    Random search for hyper-parameter opti- mization,

    J. Bergstra and Y . Bengio, “Random search for hyper-parameter opti- mization,”Journal of Machine Learning Research, vol. 13, pp. 281–305, 2012

    2012

  26. [34]

    Moot: a repository of many multi-objective optimization tasks,

    T. Menzies, T. Chen, Y . Ye, K. K. Ganguly, A. Rayegan, S. Srinivasan, and A. Lustosa, “Moot: a repository of many multi-objective optimization tasks,” in2026 IEEE/ACM 23rd International Conference on Mining Software Repositories (MSR). IEEE, 2026

    2026

  27. [35]

    Transferring performance prediction models across different hardware platforms,

    P. Valov, J.-C. Petkovich, J. Guo, S. Fischmeister, and K. Czarnecki, “Transferring performance prediction models across different hardware platforms,” inProceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, 2017, pp. 39–50

    2017

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  29. [37]

    From brittle to robust: Improving llm annotations for se optimization,

    L. Senthilkumar and T. Menzies, “From brittle to robust: Improving llm annotations for se optimization,”Empirical Software Engineering, vol. 31, no. 6, p. 155, 2026

    2026

  30. [38]

    Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement,

    S. Daulton, M. Balandat, and E. Bakshy, “Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement,” Advances in neural information processing systems, vol. 34, pp. 2187– 2200, 2021

    2021

  31. [39]

    DEHB: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization,

    N. Awad, N. Mallik, and F. Hutter, “DEHB: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21), Z.-H. Zhou, Ed. ijcai.org, 2021, pp. 2147–2153

    2021

  32. [40]

    Settles,Active learning

    B. Settles,Active learning. Morgan & Claypool Publishers, 2012

    2012

  33. [41]

    P. R. Cohen,Empirical methods for artificial intelligence. MIT press Cambridge, MA, 1995, vol. 139

    1995

  34. [42]

    “sampling

    J. Chen, V . Nair, R. Krishna, and T. Menzies, ““sampling” as a baseline optimizer for search-based software engineering,”IEEE Transactions on Software Engineering, vol. 45, no. 6, pp. 597–614, 2018

    2018

  35. [43]

    A density-based algorithm for discovering clusters in large spatial databases with noise,

    M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, ser. KDD’96. AAAI Press, 1996, pp. 226–231

    1996

  36. [44]

    Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions,

    N. Hansen, S. Finck, R. Ros, and A. Auger, “Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions,” Ph.D. dissertation, INRIA, 2009

    2009

  37. [45]

    Using simulation to investigate requirements prioritization strategies,

    D. Port, A. Olkov, and T. Menzies, “Using simulation to investigate requirements prioritization strategies,” in2008 23rd IEEE/ACM Interna- tional Conference on Automated Software Engineering. IEEE, 2008, pp. 268–277

    2008

  38. [46]

    Xomo: Understanding development options for autonomy,

    T. Menzies and J. Richardson, “Xomo: Understanding development options for autonomy,” inCOCOMO forum, vol. 2005, 2005

    2005

  39. [47]

    How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance,

    M. Li, T. Chen, and X. Yao, “How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance,”IEEE Trans. Softw. Eng., vol. 48, no. 5, p. 1771–1799, May

  40. [48]

    Available: https://doi.org/10.1109/TSE.2020.3036108

    [Online]. Available: https://doi.org/10.1109/TSE.2020.3036108

    2020 doi

  41. [49]

    A systematic review of effect size in software engineering experiments,

    V . B. Kampenes, T. Dybå, J. E. Hannay, and D. I. K. Sjøberg, “A systematic review of effect size in software engineering experiments,” Information and Software Technology, vol. 49, no. 11–12, pp. 1073–1086,

  42. [50]

    Available: https://doi.org/10.1016/j.infsof.2007.02.015

    [Online]. Available: https://doi.org/10.1016/j.infsof.2007.02.015

    2007 doi

  43. [51]

    Dominance statistics: Ordinal analyses to answer ordinal questions,

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions,”Psychological Bulletin, vol. 114, no. 3, pp. 494–509, 1993. [Online]. Available: https://doi.org/10.1037/0033-2909.114.3.494

    1993 doi

  44. [52]

    Robust confidence intervals for effect sizes: A comparative study of Cohen’s d and Cliff’s delta under non-normality and heterogeneous variances,

    M. R. Hess and J. D. Kromrey, “Robust confidence intervals for effect sizes: A comparative study of Cohen’s d and Cliff’s delta under non-normality and heterogeneous variances,” inAnnual Meeting of the American Educational Research Association, San Diego, CA, 2004. [Online]. A...

    2004

  45. [53]

    The Kolmogorov-Smirnov test for goodness of fit,

    F. J. Massey, “The Kolmogorov-Smirnov test for goodness of fit,” Journal of the American Statistical Association, vol. 46, no. 253, pp. 68–78, 1951. [Online]. Available: https://www.tandfonline.com/doi/abs/ 10.1080/01621459.1951.10500769

    1951 doi

  46. [54]

    Probable correctness theory,

    R. G. Hamlet, “Probable correctness theory,”Information processing letters, vol. 25, no. 1, pp. 17–25, 1987

    1987

  47. [55]

    New effect size rules of thumb,

    S. S. Sawilowsky, “New effect size rules of thumb,”Journal of modern applied statistical methods, vol. 8, no. 2, p. 26, 2009

    2009

  48. [56]

    C. C. McGeoch,A guide to experimental algorithmics. Cambridge University Press, 2012

    2012

  49. [57]

    Large language model-based evolutionary optimizer: Reasoning with elitism,

    S. Brahmachary, S. M. Joshi, A. Panda, K. Koneripalli, A. K. Sagotra, H. Patel, A. Sharma, A. D. Jagtap, and K. Kalyanaraman, “Large language model-based evolutionary optimizer: Reasoning with elitism,” Neurocomputing, vol. 622, p. 129272, 2025

    2025

  50. [58]

    Improving LLM-based global optimization with search space partitioning,

    A. Schwanke, L. Ivanov, D. Salinas, F. Ferreira, A. Klein, F. Hutter, and A. Zela, “Improving LLM-based global optimization with search space partitioning,” inThe Fourteenth International Conference on Learning Representations, 2026

    2026

  51. [59]

    Evolution of heuristics: towards efficient automatic algorithm design using large language model,

    F. Liu, X. Tong, M. Yuan, X. Lin, F. Luo, Z. Wang, Z. Lu, and Q. Zhang, “Evolution of heuristics: towards efficient automatic algorithm design using large language model,”Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  52. [60]

    Reevo: large language models as hyper-heuristics with reflective evolution,

    H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, “Reevo: large language models as hyper-heuristics with reflective evolution,”Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024

    2024

  53. [61]

    Search-based llms for code optimization,

    S. Gao, C. Gao, W. Gu, and M. R. Lyu, “Search-based llms for code optimization,”Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, p. 578–590, 2025. [Online]. Available: https://doi.org/10.1109/ICSE55347.2025.00021

    2025 doi

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.