Pith. sign in

REVIEW 3 major objections 6 minor 104 references

Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE

T0 review · 3 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single optimizer wins SE configuration: the best one switches with how many labels you can afford, and a free table of two cheap task attributes recovers the right choice about three times in four.

desk verdict Large, careful SBSE tournament with a usable budget-aware guide; RF surrogate is the real soft spot, not a reason to ignore the work. read the letter →

arxiv 2607.11705 v1 pith:PWEMLRYF submitted 2026-07-13 cs.SE

Kishan Kumar Ganguly , Tim Menzies This is my paper
classification cs.SE
keywords search-basedsoftwareengineeringalgorithmselectionblack-boxoptimizationlabelingbudgetconfigurationtuninginstancespacedifferentialevolutionactivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Software systems expose huge configuration spaces, but each trial can mean a full build or test run, so engineers can often afford only dozens of labels. The paper runs a large tournament of 20 black-box optimizers, grouped by the landscape assumptions they bet on, across 106 search-based software engineering tasks at four labeling budgets. It finds that no optimizer wins everywhere: the champion migrates from a geometric active learner when labels are scarce to differential evolution once more evaluations are available, and a winner chosen at one budget falls outside the top statistical tier at another on up to half the tasks. Multi-objective methods such as NSGA-II need roughly five times the budget to match what single-objective methods reach with fewer labels. Because repeating the tournament is too expensive for practice, the authors replace those 14,000 CPU hours with a zero-probe lookup keyed on whether objectives conflict, the shape of the input space, and the labeling budget; that guide ties or beats a hindsight oracle on about three-quarters of held-out tasks.

What carries the argument

An assumption-indexed tournament tree: 20 optimizers are clustered by six landscape bets (local continuity, diversification, decomposability, Pareto search, surrogates, low intrinsic dimensionality) and matched head-to-head so each internal node isolates one assumption; winners are decided by Scott-Knott clustering on distance-to-heaven and regret reduction at each budget.

What would settle it

Re-run the same tournament on a subset of the 106 tasks using real builds or test-suite scores instead of the random-forest oracle; if the budget-dependent champion migration (EZR early, DE late) or the heatmap cells reverse under ground-truth evaluation, the central claim fails.

Watch

Extended reading notes

Core claim

Across 106 SE tasks and four budgets, no optimizer dominates. The tournament champion migrates with budget from EZR (a geometric active learner) at B ≤ 50 to differential evolution at B ≥ 100, so a winner crowned at one budget is outside the target budget's top Scott-Knott tier on up to 50% of tasks. A free selection table on objective structure, input-space shape, and budget ties or beats a hindsight oracle on ≈74% of held-out tasks, while expensive landscape features reach only about 44%.

Load-bearing premise

Every ranking treats a random-forest model fitted to the full data table as the true score of a configuration, rather than real builds or runs; if that model systematically mis-scores regions some optimizers propose, the reported winners and the selection table can be wrong.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper organizes 20 black-box optimizers by six landscape assumptions, then runs an assumption-indexed tournament over 106 MOOT SE tasks at four labeling budgets (B∈{30,50,100,200}), 20 seeds, and ~14,000 CPU hours. It reports that no optimizer wins outright: the single-objective champion migrates from EZR at tight budgets to DE once labels are plentiful, so a winner crowned at one budget falls outside the target budget’s top Scott-Knott tier on up to half the tasks. Single-objective search matches or beats multi-objective methods at equal budget (with NSGA-II needing ~5× the budget to catch EZR on d2h and frontier metrics). Instance-space features fail to predict winners (~44%), but a zero-probe guide keyed on objective structure, input-space shape, and budget ties or beats a hindsight oracle on ~74% of held-out tasks under stratified CV. An open replication package is provided.

Significance. If the results hold under a more carefully validated evaluation oracle, this is a high-value empirical contribution for SBSE and configuration tuning. The scale (20×106×4×20), the explicit random-search floor and regret-reduction metric, Scott-Knott with bootstrap and Cliff’s delta, the multi-objective cross-check (IGD/GD/HV in Table VI), and the stratified CV of the guide against a hindsight oracle (Table VII) are stronger than typical optimizer leaderboards. The budget-dependent champion migration and the cheap two-attribute guide are practically actionable and falsifiable; the open tournament rig is a reusable community artifact. Credit is also due for reporting the instance-clustering failure honestly rather than forcing a feature-based narrative.

major comments (3)
  1. [Section IV-A (Evaluating a configuration)] Section IV-A: every ranking, ρ value, champion migration (EZR o DE), and heatmap cell rests on a single per-task random-forest surrogate fitted to the full MOOT table and used as the evaluation oracle. The shared-bias argument (one fixed surrogate per task) does not establish that surrogate error is uncorrelated with optimizer family. Membership-query methods (SMAC, TPE) propose points between observed rows; RF interpolation error can systematically under- or over-score those regions relative to on-row proposals from EZR, DE, or random search. The multi-objective metrics in Table VI are also computed against the same surrogate-derived reference, so they do not independently corroborate the single-vs-multi result. Please add (i) surrogate fidelity diagnostics (e.g., hold-out RMSE/R² stratified by on-row vs. interpolated queries), and (ii) a sensitivity check that re-ranks a subset of task
  2. [Table IV; RQ2 / Table V] Table IV and budgets B∈{30,50}: many population methods are run at originating-paper defaults with population size 10. At B=30 this yields only a few generations, which systematically starves GA/PSO/DE/NSGA-II relative to single-incumbent and geometric methods (EZR, hill climbing, SA). The paper correctly wants to avoid a tuning-the-tuner confound, but fixed tiny populations at the lowest budgets risk attributing wins to evaluation schedule rather than to the stated landscape assumptions (A1–A7). Please either (a) scale population/init size as a function of B so each family receives a comparable number of generations/updates, or (b) report an ablation with budget-proportional populations and show that the EZR o DE migration and Grand Final outcomes are stable. As written, RQ2’s low-budget EZR dominance may be partly an artifact of under-powered population search.
  3. [Section VI-D / Figure 5 / Table VII] RQ4 / Figure 5 and Table VII: the guide’s construction and the meaning of “tie or beat a hindsight oracle” need tighter specification. How is the recommended optimizer chosen inside each (objective structure × input-space shape × budget) cell on the training folds—Scott-Knott champion, plurality of task-level winners, or best median d2h? Does “tie” mean exact optimizer match, same Scott-Knott tier, or equal-or-better d2h on the held-out task? Without these definitions, the 74.2% figure is hard to interpret or reproduce. Please state the cell-aggregation rule and the win/tie predicate explicitly, and report exact-match accuracy separately from same-tier / equal-quality rates.
minor comments (6)
  1. [Figure 1; Abstract] Figure 1 right panel and the abstract claim “up to half our tasks”: clarify whether “best differs” means different optimizer identity or different Scott-Knott tier; the two have different practical force.
  2. [Section IV-B, Eq. (1)] Equation (1): d2h averages squared oriented objectives; state whether objectives are min-max normalized per task using the full table or only the labeled subset available at budget B (the latter would be more realistic for a practitioner).
  3. [Table III; Figure 2] Table III lists A6 (sequential reward / RL) but excludes it from the tournament; a one-sentence pointer in the Figure 2 caption that A6 is out of scope at few-dozen budgets would help readers who only skim the tree.
  4. [Section VI-D.2] The large-numeric threshold ∑ log2(|Xi|)≥40 is called “empirically defined”; the sensitivity check (10–60) is welcome—please put the per-threshold overall win rates in a short appendix table rather than only the ≈6% drop summary.
  5. [Table I; Table II; Figure 2] Minor typos / consistency: “SW AY” spacing in Table II and Figure 2; “TheseCORE OF MOOT” in Table I caption; “ϵ-pruning” vs “ϵ-bin” for DODGE; arXiv id and venue placeholders (“to appear”) should be cleaned for camera-ready.
  6. [Figure 3] Figure 3 shows runtimes only for a subset of winning optimizers; a full 20-optimizer cost table (median wall-clock per budget) in the appendix would better support the claim that the tournament is impractical for practitioners to re-run.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical tournament with held-out CV guide; no derivation reduces to its inputs by construction.

full rationale

The paper is an empirical algorithm-selection study, not a first-principles derivation. Tournament winners are produced by 20-seed Scott-Knott ranking of d2h (and frontier metrics) under a fixed per-task RF surrogate; the selection guide is then built only on train folds and scored against a hindsight oracle on held-out tasks under stratified CV, so the 74.2% figure is not tautological. Instance-clustering failure (44.2%) is reported rather than suppressed. Self-citations to prior MOOT/EZR/SWAY/DODGE work supply tasks and methods but do not force the budget-migration or guide results. The RF-oracle premise is a validity threat (shared bias may still correlate with optimizer family), not circularity: rankings are not definitionally equal to the surrogate fit. No self-definitional equations, fitted-input-called-prediction, uniqueness theorems, or ansatz-smuggling steps appear. Score 1 for ordinary non-load-bearing self-citation only.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The central claims rest on standard statistical practice plus several domain modeling choices: RF surrogates as oracles, d2h as the scalar quality, paper-default hyperparameters, a hand-chosen search-space-size threshold, and the premise that MOOT tabular tasks represent budgeted SE black-box optimization. No new physical entities are postulated; free parameters are methodological cutoffs and defaults rather than fitted physical constants. The guide’s accuracy is an empirical measurement under CV, not a derivation from those parameters alone.

free parameters (5)
  • large-numeric search-space threshold
    Tasks with Σ log2(|Xi|) ≥ 40 are labeled large-numeric; boundary chosen empirically with sensitivity check over {10…60} (Section VI-D). Affects heatmap cell assignment and thus recommended optimizer.
  • binary/SAT column fraction
    ≥80% of decision variables binary classifies a task as Binary/SAT (Section VI-D). Hand threshold that routes tasks into different guide cells.
  • evaluation budgets B ∈ {30,50,100,200}
    Discrete budget grid chosen by authors; all migration and guide results are defined on this grid. Continuous-budget behavior is unmeasured.
  • optimizer population / default hyperparameters
    Population 10, DE F=0.8 CR=0.9, tabu tenure 7, EZR init labels 4, etc. (Table IV) taken from originating papers; not re-tuned. Relative ranks could shift under fair per-task tuning.
  • Scott-Knott / Cliff’s delta effect-size gate
    Statistical grouping procedure and non-negligible effect-size threshold determine which methods ‘win’ a match; different gates could re-tier close contests.
assumptions (6)
  • domain assumption A per-task random-forest ensemble fitted on all training rows is a fair shared oracle for scoring candidate configurations, including points between observed rows.
    Section IV-A; load-bearing for every d2h, ρ, and ranking. Justified by categorical/numeric SE data and multi-branch landscapes, but not validated against live builds in this study.
  • domain assumption Normalized Euclidean distance-to-heaven (d2h) is an adequate scalar quality for ranking both single- and multi-objective methods; best-point reduction of a returned set is what a practitioner ships.
    Equation (1) and Section IV-B; partially stress-tested with IGD/GD/HV in RQ3 but remains a modeling choice that could favor scalar methods.
  • standard math No-Free-Lunch: no optimizer dominates all tasks; therefore assumption-indexed sampling of representatives is a valid way to organize the menu.
    Cited Wolpert & Macready; used in Sections I–III to justify clustering rather than exhaustive comparison.
  • domain assumption Landscape assumptions A1–A7 (local continuity, diversification, building-block decomposability, Pareto incomparability, surrogate feasibility, sequential reward, low intrinsic dimensionality) correctly partition the chosen optimizers.
    Table III and Section III; read from originating papers. Mis-assignment of an optimizer to an assumption would misread ‘why’ a branch won.
  • ad hoc to paper Running every method at originating-paper defaults avoids a tuning-the-tuner confound and attributes wins to assumptions rather than hyperparameter skill.
    Section IV / Table IV. Contestable: defaults may systematically favor some families on SE data.
  • domain assumption MOOT’s 106 SE tasks (configuration, performance, cloud, project health, feature models, process/cost, testing) are representative enough for external claims about SBSE optimizer selection.
    Table I and threats (Section VI-E). Standard external-validity premise of the study.
invented entities (2)
  • Assumption-indexed tournament tree (Fig. 2)
    purpose: Organizes 20 optimizers into head-to-head matches that isolate one landscape assumption per branch, culminating in SO vs MO grand final.
    Methodological scaffold invented for this study; not an external physical entity. Independent evidence is the empirical match outcomes themselves.
  • Budget-aware two-attribute heatmap guide (Fig. 5) independent evidence
    purpose: Maps (objective structure × input-space shape × budget) to a recommended optimizer without probes.
    Primary practical artifact. Validated under stratified CV against hindsight oracle; still a construct fitted to this corpus’s cell winners.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE." pith.science (2026). https://pith.science/paper/PWEMLRYF

@misc{pith2026260711705,
  author       = {Pith},
  title        = {Pith review of: Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PWEMLRYF}},
  note         = {Machine review of arXiv:2607.11705}
}
abstract

Configuring and tuning modern software is unavoidable, expensive, and error-prone: a single system can expose hundreds of interacting options, and scoring one setting can mean a full build or test run. The standard response is automated optimization, but the number of available optimizers is large and growing. And some of the guidance for selecting among them is misleading: NSGA-II, for example, is widely recommended, yet other algorithms reach the same results using only 1/20th as many evaluations. To help practitioners make better choices about tools to configure their systems, we cluster 20 optimizers, based on six assumptions about the data. Next, we run a tournament across those optimizers, using 106 SE optimization tasks at four labeling budgets (taking 14,000+ CPU hours). We find that no optimizer wins outright. The best one migrates with the budget (from a geometric active learner when labels are scarce to differential evolution when labels are plentiful) so a winner "crowned" at one budget is wrong at another on up to half our tasks. Running such a tournament for every new domain is impractical due to its CPU cost. Fortunately, we find that those 14,000 hours can be replaced by a table lookup over two cheap-to-obtain task attributes (plus the labeling budget). Predictions from this table tie or beat a hindsight oracle on $\approx 75%$ of held-out tasks. To support open science, our tournament and replication package are open-sourced for SBSE researchers and practitioners at https://github.com/KKGanguly/OptimizerTournament.

Figures

Figures reproduced from arXiv: 2607.11705 by the authors.

Figure 1
Figure 1. Optimization can be very effective for SE tasks. Left panel: In 106 MOOT SE tasks, reasoning on just 30 labels already closes [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The assumption-indexed tournament tree (20 optimizers). Leaves are [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Search cost is real and it is the reason a free guide matters. Per-run wall-clock distributions for every wining optimizer at each budget, aggregated to the ≈14,000 CPU hours the full study consumed. optimizers’ results [42], [43], [75]. Lower GD and IGD and higher HV are better. These are reported only as a cross-check; the tournament itself is decided on d2h. Statistics. Every match and every ranking uses the same… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Static instance clustering features cam not find the right optimizer A two-dimensional projection of the tasks in landscape-feature space, colored by the budget-200 winning family. If cheap features predicted the winner, the colors would separate into clean footprints.…
Figure 5
Figure 5. Figure 5: shows the resulting guide, a recommended op￾timizer per attribute cell per budget. EZR owns the single￾objective small-numeric cells at tight budgets; DE and SMAC take over as the input space grows or the budget increases, exactly the migration RQ2 exposed. B=30 B=50 B…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

104 extracted references · 2 linked inside Pith

  1. [1]

    Minimal data, maximum clarity: A heuristic for explaining optimization,

    A. Rayegan and T. Menzies, “Minimal data, maximum clarity: A heuristic for explaining optimization,”Journal of Systems and Software, p. 112897, 2026

    2026

  2. [2]

    Moot: a repository of many multi-objective optimiza- tion tasks,

    T. Menzies, T. Chen, Y . Ye, K. K. Ganguly, A. Rayegan, S. Srinivasan, and A. Lustosa, “Moot: a repository of many multi-objective optimiza- tion tasks,” inProc. 22nd International Conference on Mining Software Repositories, Data and Tool Showcase Track, 2026, to appear

    2026

  3. [3]

    Promisetune: Unveiling causally promising and explainable configuration tuning,

    P. Chen and T. Chen, “Promisetune: Unveiling causally promising and explainable configuration tuning,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026, to appear

    2026

  4. [4]

    Can large language models improve se active learning via warm-starts?

    L. Senthilkumar and T. Menzies, “Can large language models improve se active learning via warm-starts?”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  5. [5]

    Less noise, more signal: Drr for better optimizations of se tasks,

    A. Lustosa and T. Menzies, “Less noise, more signal: Drr for better optimizations of se tasks,”arXiv preprint arXiv:2503.21086, 2025

    arXiv 2025

  6. [6]

    Learning from very little data: On the value of landscape analysis for predicting software project health,

    ——, “Learning from very little data: On the value of landscape analysis for predicting software project health,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, pp. 1–22, 2024

    2024

  7. [7]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,”IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2018

    2018

  8. [8]

    Russell and P

    S. Russell and P. Norvig,Artificial Intelligence: A Modern Approach, 4/E.Pearson, 2021

    2021

Show all 104 references
  1. [9]

    A theoretical and empirical study of search-based testing: Local, global, and hybrid search,

    M. Harman and P. McMinn, “A theoretical and empirical study of search-based testing: Local, global, and hybrid search,”IEEE Transac- tions on Software Engineering, vol. 36, no. 2, pp. 226–247, 2009

    2009

  2. [10]

    Search-based software engineering: Trends, techniques and applications,

    M. Harman, S. A. Mansouri, and Y . Zhang, “Search-based software engineering: Trends, techniques and applications,”ACM Computing Surveys (CSUR), vol. 45, no. 1, pp. 1–61, 2012

    2012

  3. [11]

    Pyart: Python api recommendation in real-time,

    X. He, L. Xu, X. Zhang, R. Hao, Y . Feng, and B. Xu, “Pyart: Python api recommendation in real-time,” inIEEE/ACM 43rd International Conference on Software Engineering. IEEE, 2021, pp. 1634–1645

    2021

  4. [12]

    Optimization by simulated annealing,

    S. Kirkpatrick, C. D. Gelatt Jr, and M. P. Vecchi, “Optimization by simulated annealing,”science, vol. 220, no. 4598, pp. 671–680, 1983

    1983

  5. [13]

    The current state and future of search based software engineering,

    M. Harman, “The current state and future of search based software engineering,” inFOSE’07. IEEE, 2007, pp. 342–357

    2007

  6. [14]

    Search-based fault local- ization,

    S. Wang, D. Lo, L. Jiang, H. C. Lauet al., “Search-based fault local- ization,” in26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). IEEE, 2011, pp. 556–559

    2011

  7. [15]

    Evolutionsstrategie,

    I. Rechenberg, “Evolutionsstrategie,”Optimierung technischer Systeme nach Prinzipien derbiologischen Evolution, 1973

    1973

  8. [16]

    Test suite generation with the many independent objective (mio) algorithm,

    A. Arcuri, “Test suite generation with the many independent objective (mio) algorithm,”Inf. Softw. Technol., vol. 104, pp. 195–206, 2018

    2018

  9. [17]

    Iterated local search,

    H. R. Lourenc ¸o, O. C. Martin, and T. St ¨utzle, “Iterated local search,” inHandbook of metaheuristics. Springer, 2003, pp. 320–353

    2003

  10. [18]

    A genetic programming based iterated local search for software project scheduling,

    N. R. Sabar, A. Turky, and A. Song, “A genetic programming based iterated local search for software project scheduling,” inProc. Genetic and Evolutionary Computation Conference, 2018, pp. 1364–1370

    2018

  11. [19]

    Tabu search—part i,

    F. Glover, “Tabu search—part i,”ORSA Journal on computing, vol. 1, no. 3, pp. 190–206, 1989

    1989

  12. [20]

    A tabu search algorithm for structural software testing,

    E. D ´ıaz, J. Tuya, R. Blanco, and J. J. Dolado, “A tabu search algorithm for structural software testing,”Computers & Operations Research, vol. 35, no. 10, pp. 3052–3072, 2008

    2008

  13. [21]

    Transfer learning for cross- company software defect prediction,

    Y . Ma, G. Luo, X. Zeng, and A. Chen, “Transfer learning for cross- company software defect prediction,”Information and Software Tech- nology, vol. 54, no. 3, pp. 248–256, 2012

    2012

  14. [22]

    How to “dodge

    A. Agrawal, W. Fu, D. Chen, X. Shen, and T. Menzies, “How to “dodge” complex software analytics,”IEEE Transactions on Software Engineering, vol. 47, no. 10, pp. 2182–2194, 2019

    2019

  15. [23]

    J. H. Holland,Adaptation in natural and artificial systems: an intro- ductory analysis with applications to biology, control, and artificial intelligence. MIT press, 1992

    1992

  16. [24]

    Genprog: A generic method for automatic software repair,

    C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,”IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 54–72, 2011

    2011

  17. [25]

    An empirical study of meta-and hyper-heuristic search for multi-objective release planning,

    Y . Zhang, M. Harman, G. Ochoa, G. Ruhe, and S. Brinkkemper, “An empirical study of meta-and hyper-heuristic search for multi-objective release planning,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 27, no. 1, pp. 1–32, 2018

    2018

  18. [26]

    From recombination of genes to the estimation of distributions i. binary parameters,

    H. M ¨uhlenbein and G. Paass, “From recombination of genes to the estimation of distributions i. binary parameters,” inInternational conference on parallel problem solving from nature. Springer, 1996, pp. 178–187

    1996

  19. [27]

    Test data generation for mutation testing based on markov chain usage model and estimation of distribution algorithm,

    C. Wei, X. Yao, D. Gong, and H. Liu, “Test data generation for mutation testing based on markov chain usage model and estimation of distribution algorithm,”IEEE Transactions on Software Engineering, vol. 50, no. 3, pp. 551–573, 2024

    2024

  20. [28]

    Particle swarm optimization,

    J. Kennedy and R. Eberhart, “Particle swarm optimization,” inProceed- ings of ICNN’95-international conference on neural networks, vol. 4. ieee, 1995, pp. 1942–1948

    1995

  21. [29]

    Learning seed-adaptive mutation strategies for greybox fuzzing,

    M. Lee, S. Cha, and H. Oh, “Learning seed-adaptive mutation strategies for greybox fuzzing,” in2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE). IEEE, 2023, pp. 384–396

    2023

  22. [30]

    Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,

    R. Storn and K. Price, “Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,”Journal of global optimization, vol. 11, no. 4, pp. 341–359, 1997

    1997

  23. [31]

    Easy over hard: A case study on deep learning,

    W. Fu and T. Menzies, “Easy over hard: A case study on deep learning,” inProceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 49–60

    2017

  24. [32]

    Sequential model-based optimization for general algorithm configuration,

    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” inLION. Springer, 2011, pp. 507–523

    2011

  25. [33]

    How low can you go? the data-light SE challenge,

    K. K. Ganguly and T. Menzies, “How low can you go? the data-light SE challenge,” inProceedings of the ACM International Conference on the Foundations of Software Engineering (FSE), 2026, to appear

    2026

  26. [34]

    Algorithms for hyper-parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. K ´egl, “Algorithms for hyper-parameter optimization,”NeurIPS, vol. 24, 2011

    2011

  27. [35]

    Efficient compiler autotuning via bayesian optimization,

    J. Chen, N. Xu, P. Chen, and H. Zhang, “Efficient compiler autotuning via bayesian optimization,” inIEEE/ACM 43rd International Confer- ence on Software Engineering (ICSE). IEEE, 2021, pp. 1198–1209

    2021

  28. [36]

    “sampling

    J. Chen, V . Nair, R. Krishna, and T. Menzies, ““sampling” as a baseline optimizer for search-based software engineering,”IEEE Transactions on Software Engineering, vol. 45, no. 6, pp. 597–614, 2018

    2018

  29. [37]

    Accuracy can lie: On the impact of surrogate model in configuration tuning,

    P. Chen, J. Gong, and T. Chen, “Accuracy can lie: On the impact of surrogate model in configuration tuning,”IEEE Transactions on Software Engineering, vol. 51, no. 2, pp. 548–580, 2025

    2025

  30. [38]

    Random search for hyper-parameter opti- mization

    J. Bergstra and Y . Bengio, “Random search for hyper-parameter opti- mization.”Journal of machine learning research, vol. 13, no. 2, 2012

    2012

  31. [39]

    A fast and elitist multiobjective genetic algorithm: Nsga-ii,

    K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,”IEEE transactions on evolutionary computation, vol. 6, no. 2, pp. 182–197, 2002

    2002

  32. [40]

    Many-objective software remodularization using nsga-iii,

    W. Mkaouer, M. Kessentini, A. Shaout, P. Koligheu, S. Bechikh, K. Deb, and A. Ouni, “Many-objective software remodularization using nsga-iii,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 24, no. 3, pp. 1–45, 2015

    2015

  33. [41]

    A multi-objective test data generation approach for mutation testing of feature models,

    R. A. Matnei Filho and S. R. Vergilio, “A multi-objective test data generation approach for mutation testing of feature models,”J. softw. eng. res. dev., vol. 4, no. 1, p. 4, 2016

    2016

  34. [42]

    Spea2: Improving the strength pareto evolutionary algorithm,

    E. Zitzler, M. Laumanns, and L. Thiele, “Spea2: Improving the strength pareto evolutionary algorithm,”TIK report, vol. 103, 2001

    2001

  35. [43]

    Sms-emoa: Multiobjective selection based on dominated hypervolume,

    N. Beume, B. Naujoks, and M. Emmerich, “Sms-emoa: Multiobjective selection based on dominated hypervolume,”European journal of operational research, vol. 181, no. 3, pp. 1653–1669, 2007

    2007

  36. [44]

    An empirical study on pareto based multi-objective feature selection for software defect prediction,

    C. Ni, X. Chen, F. Wu, Y . Shen, and Q. Gu, “An empirical study on pareto based multi-objective feature selection for software defect prediction,”J. Syst. Softw., vol. 152, pp. 215–238, 2019

    2019

  37. [45]

    Moea/d: A multiobjective evolutionary algorithm based on decomposition,

    Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,”IEEE Transactions on evolutionary compu- tation, vol. 11, no. 6, pp. 712–731, 2007

    2007

  38. [46]

    Compiler auto-tuning via critical flag selection,

    M. Zhu and D. Hao, “Compiler auto-tuning via critical flag selection,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1000–1011

    2023

  39. [47]

    Automatic database management system tuning through large-scale machine learn- ing,

    D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang, “Automatic database management system tuning through large-scale machine learn- ing,” inProceedings of the 2017 ACM international conference on management of data, 2017, pp. 1009–1024

    2017

  40. [48]

    On the value of user preferences in search-based software engineering: A case study in software product lines,

    A. S. Sayyad, T. Menzies, and H. Ammar, “On the value of user preferences in search-based software engineering: A case study in software product lines,” in2013 35Th international conference on software engineering (ICSE). IEEE, 2013, pp. 492–501

    2013

  41. [49]

    An end-to-end automatic cloud database tuning system using deep reinforcement learning,

    J. Zhang, Y . Liu, K. Zhou, G. Li, Z. Xiao, B. Cheng, J. Xing, Y . Wang, T. Cheng, L. Liuet al., “An end-to-end automatic cloud database tuning system using deep reinforcement learning,” inProceedings of the 2019 international conference on management of data, 2019, pp. 415–432

    2019

  42. [50]

    Iterative generation of adversarial example for deep code models,

    L. Huang, W. Sun, and M. Yan, “Iterative generation of adversarial example for deep code models,” inIEEE/ACM 47th International Conference on Software Engineering. IEEE, 2025, pp. 2213–2224

    2025

  43. [51]

    No free lunch theorems for optimization,

    D. H. Wolpert and W. G. Macready, “No free lunch theorems for optimization,”IEEE transactions on evolutionary computation, vol. 1, no. 1, pp. 67–82, 2002

    2002

  44. [52]

    Toward automated algorithm design: A survey and practical guide to meta- black-box-optimization,

    Z. Ma, H. Guo, Y .-J. Gong, J. Zhang, and K. C. Tan, “Toward automated algorithm design: A survey and practical guide to meta- black-box-optimization,”IEEE Trans. Evol. Comput., 2025

    2025

  45. [53]

    How efficient is llm-generated code? a rigorous & high-standard benchmark,

    R. Qiu, W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 2233–2261

    2025

  46. [54]

    Simpler hyperparameter optimization for software analytics: Why, how, when?

    A. Agrawal, X. Yang, R. Agrawal, R. Yedida, X. Shen, and T. Menzies, “Simpler hyperparameter optimization for software analytics: Why, how, when?”IEEE Transactions on Software Engineering, vol. 48, no. 8, pp. 2939–2954, 2021

    2021

  47. [55]

    Analysing the fitness landscape of search-based software testing problems,

    A. Aleti, I. Moser, and L. Grunske, “Analysing the fitness landscape of search-based software testing problems,”Automated Software Engi- neering, vol. 24, no. 3, pp. 603–621, 2017

    2017

  48. [56]

    Causes and effects of fitness landscapes in unit test generation,

    N. Albunian, G. Fraser, and D. Sudholt, “Causes and effects of fitness landscapes in unit test generation,” inProceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020, pp. 1204–1212

    2020

  49. [57]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    2024

  50. [58]

    The design, analysis and interpretation of reper- tory grids,

    M. Easterby-Smith, “The design, analysis and interpretation of reper- tory grids,”Int. J. Man-Mach. Stud., vol. 13, no. 1, pp. 3–24, 1980

    1980

  51. [59]

    Heuristics for systems engineering cost estimation,

    R. Valerdi, “Heuristics for systems engineering cost estimation,”IEEE Systems Journal, vol. 5, no. 1, pp. 91–98, 2010

    2010

  52. [60]

    Identifying self-admitted technical debts with jitterbug: A two-step approach,

    Z. Yu, F. M. Fahid, H. Tu, and T. Menzies, “Identifying self-admitted technical debts with jitterbug: A two-step approach,”IEEE Transac- tions on Software Engineering, vol. 48, no. 5, pp. 1676–1691, 2020

    2020

  53. [61]

    Data quality matters: A case study on data label correctness for security bug report prediction,

    X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,”IEEE Trans. Softw. Eng., vol. 48, no. 7, pp. 2541–2556, 2021

    2021

  54. [62]

    Detecting false alarms from automatic static analysis tools: How far are we?

    H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” inProc. of the 44th International Conference on Software Engineering, 2022, pp. 698–709

    2022

  55. [63]

    Data quality: Some comments on the nasa software defect datasets,

    M. Shepperd, Q. Song, Z. Sun, and C. Mair, “Data quality: Some comments on the nasa software defect datasets,”IEEE Transactions on software engineering, vol. 39, no. 9, pp. 1208–1215, 2013

    2013

  56. [64]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,”IEEE Trans. Softw. Eng., vol. 39, no. 6, pp. 757–773, 2012

    2012

  57. [65]

    Can llms re- place manual annotation of software engineering artifacts?

    T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms re- place manual annotation of software engineering artifacts?” in2025 IEEE/ACM 22nd International Conference on Mining Software Repos- itories (MSR). IEEE, 2025, pp. 526–538

    2025

  58. [66]

    Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo,

    K. Eggensperger, P. M ¨uller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, and F. Hutter, “Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo,” inThirty- fifth Conference on Neural Information Processing Systems Datasets and Benc...

    2021

  59. [67]

    Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation,

    A. I. Cowen-Rivers, W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Maraval, H. Jianye, J. Wang, J. Peterset al., “Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation,” J. Artif. Intell. Res., vol. 74, pp. 1269–1349, 2022

    2022

  60. [68]

    Scalable global optimization via local bayesian optimization,

    D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek, “Scalable global optimization via local bayesian optimization,”Ad- vances in neural information processing systems, vol. 32, 2019

    2019

  61. [69]

    Unicorn: Reasoning about configurable system performance through the lens of causality,

    M. S. Iqbal, R. Krishna, M. A. Javidian, B. Ray, and P. Jamshidi, “Unicorn: Reasoning about configurable system performance through the lens of causality,” inProceedings of the Seventeenth European Conference on Computer Systems, 2022, pp. 199–217

    2022

  62. [70]

    Llamatune: sample-efficient dbms configuration tuning,

    K. Kanellis, C. Ding, B. Kroth, A. M ¨uller, C. Curino, and S. Venkataraman, “Llamatune: sample-efficient dbms configuration tuning,”Proc. VLDB Endow., vol. 15, no. 11, p. 2953–2965, Jul

  63. [71]

    Available: https://doi.org/10.14778/3551793.3551844

    [Online]. Available: https://doi.org/10.14778/3551793.3551844

    doi

  64. [72]

    Qtune: A query-aware database tuning system with deep reinforcement learning,

    G. Li, X. Zhou, S. Li, and B. Gao, “Qtune: A query-aware database tuning system with deep reinforcement learning,”Proceedings of the VLDB Endowment, vol. 12, no. 12, pp. 2118–2130, 2019

    2019

  65. [73]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawziet al., “Mathematical discoveries from program search with large language models,”Nature, vol. 625, no. 7995, pp. 468–475, 2024

    2024

  66. [74]

    Reevo: Large language models as hyper-heuristics with reflective evolution,

    H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, “Reevo: Large language models as hyper-heuristics with reflective evolution,”NeurIPS, vol. 37, pp. 43 571–43 608, 2024

    2024

  67. [75]

    Genetic programming as a means for programming computers by natural selection,

    J. R. Koza, “Genetic programming as a means for programming computers by natural selection,”Statistics and computing, vol. 4, no. 2, pp. 87–112, 1994

    1994

  68. [76]

    An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,

    K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,”IEEE transactions on evolutionary computation, vol. 18, no. 4, pp. 577–601, 2013

    2013

  69. [77]

    The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering,

    T. Chen and M. Li, “The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 1, pp. 1–40, 2023

    2023

  70. [78]

    Mmo: meta multi-objectivization for software configuration tuning,

    P. Chen, T. Chen, and M. Li, “Mmo: meta multi-objectivization for software configuration tuning,”IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1478–1504, 2024

    2024

  71. [79]

    Practical bayesian optimiza- tion of machine learning algorithms,

    J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimiza- tion of machine learning algorithms,”Advances in neural information processing systems, vol. 25, 2012

    2012

  72. [80]

    Etune: Efficient con- figuration tuning for big-data software systems via configuration space reduction,

    R. Cao, L. Bao, K. Zhao, and P. Zhangsun, “Etune: Efficient con- figuration tuning for big-data software systems via configuration space reduction,”Journal of Systems and Software, vol. 209, p. 111936, 2024

    2024

  73. [81]

    Dually hierarchical drift adaptation for online configuration performance learning,

    Z. Xiang, J. Gong, and T. Chen, “Dually hierarchical drift adaptation for online configuration performance learning,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026, to appear

    2026

  74. [82]

    Cotune: Co-evolutionary configuration tuning,

    G. Xiong and T. Chen, “Cotune: Co-evolutionary configuration tuning,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, 2025, p. 1490–1502. [Online]. Available: https://doi.org/10.1109/ASE63991.2025.00126

    2025 doi

  75. [83]

    Compiler autotuning through multiple- phase learning,

    M. Zhu, D. Hao, and J. Chen, “Compiler autotuning through multiple- phase learning,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 4, pp. 1–38, 2024

    2024

  76. [84]

    Unveiling many faces of surrogate models for configuration tuning: A fitness landscape analysis perspec- tive,

    P. Chen, H. Liang, and T. Chen, “Unveiling many faces of surrogate models for configuration tuning: A fitness landscape analysis perspec- tive,”arXiv preprint arXiv:2509.21945, 2025

    2025

  77. [85]

    Learning from delayed rewards,

    B. J. Kr ¨ose, “Learning from delayed rewards,”Robotics and Au- tonomous Systems, vol. 15, no. 4, pp. 233–235, 1995

    1995

  78. [86]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  79. [87]

    Instance space analysis for algorithm testing: Methodology and software tools,

    K. Smith-Miles and M. A. Mu ˜noz, “Instance space analysis for algorithm testing: Methodology and software tools,”ACM Computing Surveys, vol. 55, no. 12, pp. 1–31, 2023

    2023

  80. [88]

    Instance space analysis of search-based software testing,

    N. Neelofar, K. Smith-Miles, M. A. Mu ˜noz, and A. Aleti, “Instance space analysis of search-based software testing,”IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2642–2660, 2022

    2022

  81. [89]

    Bohb: Robust and efficient hyperparameter optimization at scale,

    S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” inInternational conference on machine learning. PMLR, 2018, pp. 1437–1446

    2018

  82. [90]

    DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization,

    N. Awad, N. Mallik, and F. Hutter, “DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Z. Zhou, Ed. ijcai.org, 2021, pp. 2147–2153

    2021

  83. [91]

    Using large language models for hyperparameter optimization,

    M. R. Zhang, N. Desai, J. Bae, J. Lorraine, and J. Ba, “Using large language models for hyperparameter optimization,”arXiv preprint arXiv:2312.04528, 2023

    2023 arXiv

  84. [92]

    Language model crossover: Variation through few-shot prompting,

    E. Meyerson, M. J. Nelson, H. Bradley, A. Gaier, A. Moradi, A. K. Hoover, and J. Lehman, “Language model crossover: Variation through few-shot prompting,”ACM Transactions on Evolutionary Learning, vol. 4, no. 4, pp. 1–40, 2024

    2024

  85. [93]

    Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization,

    F. Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl, “Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization,” inInternational Conference on Auto- mated Machine Learning. PMLR, 2022, pp. 3–1

    2022

  86. [94]

    Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,

    A. Zela, J. N. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, “Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,” inInternational Conference on Learning Representations, 2020. [Online]. Available: https://api.semanticscholar.o...

    2020

  87. [95]

    Efficient benchmarking of hyperparameter optimizers via surrogates,

    K. Eggensperger, F. Hutter, H. Hoos, and K. Leyton-Brown, “Efficient benchmarking of hyperparameter optimizers via surrogates,” inProc. of the AAAI conference on artificial intelligence, vol. 29, no. 1, 2015

    2015

  88. [96]

    Efficient benchmarking of algorithm configurators via model- based surrogates,

    K. Eggensperger, M. Lindauer, H. H. Hoos, F. Hutter, and K. Leyton- Brown, “Efficient benchmarking of algorithm configurators via model- based surrogates,”Machine Learning, vol. 107, no. 1, pp. 15–41, 2018

    2018

  89. [97]

    A cluster analysis method for grouping means in the analysis of variance,

    A. J. Scott and M. Knott, “A cluster analysis method for grouping means in the analysis of variance,”Biometrics, pp. 507–512, 1974

    1974

  90. [98]

    Exploratory landscape analysis,

    O. Mersmann, B. Bischl, H. Trautmann, M. Preuss, C. Weihs, and G. Rudolph, “Exploratory landscape analysis,” inProc. 13th annual GECCO, 2011, pp. 829–836

    2011

  91. [99]

    Detecting funnel structures by means of exploratory landscape analysis,

    P. Kerschke, M. Preuss, S. Wessing, and H. Trautmann, “Detecting funnel structures by means of exploratory landscape analysis,” inPro- ceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, 2015, pp. 265–272

    2015

  92. [100]

    Fitness distance correlation as a measure of problem difficulty for genetic algorithms

    T. Jones, S. Forrestet al., “Fitness distance correlation as a measure of problem difficulty for genetic algorithms.” inICGA, vol. 95, 1995, pp. 184–192

    1995

  93. [101]

    The dispersion metric and the cma evolution strategy,

    M. Lunacek and D. Whitley, “The dispersion metric and the cma evolution strategy,” inProceedings of the 8th annual conference on Genetic and evolutionary computation, 2006, pp. 477–484

    2006

  94. [102]

    Automated algorithm selection: Survey and perspectives,

    P. Kerschke, H. H. Hoos, F. Neumann, and H. Trautmann, “Automated algorithm selection: Survey and perspectives,”Evolutionary computa- tion, vol. 27, no. 1, pp. 3–45, 2019

    2019

  95. [103]

    A study of cross-validation and bootstrap for accuracy estimation and model selection,

    R. Kohaviet al., “A study of cross-validation and bootstrap for accuracy estimation and model selection,” inIjcai, vol. 14, no. 2. Montreal, Canada, 1995, pp. 1137–1145

    1995

  96. [104]

    Hyperband: A novel bandit-based approach to hyperparameter opti- mization,

    L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter opti- mization,”Journal of machine learning research, vol. 18, no. 185, pp. 1–52, 2018

    2018

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.