Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Proactively tuning under a cheap, similar environment beats direct tuning of the target, saving hours of budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:19 UTC pith:CQ5ZABAA

load-bearing objection A promising, well-evidenced adaptation of multi-fidelity optimization to configurable systems, but the headline results are a bit overstated and the fidelity-perfection estimate relies on a very small sample. the 4 major comments →

arxiv 2608.00759 v1 pith:CQ5ZABAA submitted 2026-08-01 cs.SE cs.AIcs.DBcs.PF

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

Yulong Ye , Miqing Li , Tao Chen This is my paper
classification cs.SE cs.AIcs.DBcs.PF
keywords configurable systemsconfiguration tuningmulti-fidelity optimizationimperfect fidelitybudget utilizationsearch-based software engineeringhyperparameter optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Highly configurable systems are hard to tune because measuring a single configuration can take minutes to hours, so the tuning budget is quickly exhausted. This paper claims that 'less can be more': deliberately spending part of the budget in a cheaper, similar environment (an 'imperfect-fidelity' setting) can produce better final configurations for the real target environment than tuning there directly. The proposed method, MFTune, searches over thousands of possible cheap settings to find one that ranks configurations almost like the target but at a fraction of the cost, then uses it to seed a wide exploration whose best candidates are transferred to the expensive target for deep refinement. On six real systems, MFTune beat ten state-of-the-art tuners in 83.33% of comparisons, with up to 19.34% improvement and budget savings of hours. If the claim holds, it reframes budget utilization in configuration tuning: the best use of a limited measurement budget may be indirect.

Core claim

The paper's central discovery is that the conventional view of multi-fidelity optimization—where a single resource factor (like training epochs) monotonically controls both cost and accuracy—does not hold for configurable systems, and that exploiting this structural difference pays off. MFTune formalizes a fidelity setting as a vector of exogenous environment factors (budget, workload, dataset), measures its 'fidelity perfection' as the Spearman rank correlation between configurations measured there and under the target perfect-fidelity setting, and treats finding a cheap-but-faithful setting as a multi-objective search (NSGA-II) over a fidelity space of more than 10^4 candidates. The discov

What carries the argument

The central machinery is the fidelity framework plus the three-phase MFTune pipeline. An imperfect-fidelity setting is defined by a vector of exogenous fidelity factors (e.g., benchmark duration, table size, thread count) that make measurement cheaper but only approximately faithful to the target 'perfect-fidelity' environment. Fidelity perfection is measured by the Spearman rank correlation between configuration performances under the candidate setting and the target. MFTune first runs NSGA-II over the fidelity space to find a 'fair' setting that maximizes this correlation while minimizing cost, then performs diversity-preserving two-stage tuning under that cheap setting to build an archive

Load-bearing premise

The whole approach rests on the estimate that a candidate cheap environment's faithfulness can be judged from just 10 sampled configurations; if those 10 misrepresent how configuration rankings behave in a space that can be as large as 10^724, MFTune may commit to a misleading imperfect-fidelity setting and the benefits evaporate.

What would settle it

Measure the Spearman correlation between a candidate cheap setting and the perfect setting on an independent set of a few hundred configurations; if the rank correlation computed on l=10 configurations consistently disagrees with this larger-sample estimate in sign or magnitude, then the fidelity-discovery step is unreliable. A direct experiment would run MFTune while replacing the l=10 estimate with a larger verification sample and show that the chosen fair setting changes and results deteriorate.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Tuning budgets can be spent indirectly: allocating up to half the budget to a cheaper, similar environment can outperform spending the entire budget on the target environment.
  • Multi-fidelity tuning for configurable systems should not assume monotonic cost-to-accuracy; the paper shows cheaper settings sometimes have higher fidelity perfection than more expensive ones, so fidelity must be measured, not assumed.
  • The ablation study indicates that neither random fidelity selection nor single-stage imperfect-fidelity tuning suffices; both the NSGA-II discovery and the diversity-preserving two-stage seeding contribute to MFTune's superiority.
  • Multi-fidelity tuners designed for hyperparameter optimization (Hyperband, BOHB, DEHB) rank poorly because they assume cost and fidelity perfection move together, an assumption this paper shows is violated on real systems.
  • Even when MFTune does not rank first (Clang), it still achieves comparable results, and in most comparisons it reaches or beats the counterpart's best with hours to spare.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the fidelity-discovery phase consumes only a quarter of the budget and produces a reusable archive, the same fair setting could be cached and reused across multiple tuning campaigns under the same target environment; the paper does not test this reuse directly.
  • The Spearman-based fidelity perfection with l=10 sampled configurations is the load-bearing estimate; a natural extension is to make l adaptive or to confirm the ranking with a small verification set before committing to an imperfect-fidelity setting.
  • The framework of fidelity factors (budget, workload, dataset) is general enough that the same imperfect-fidelity discovery could be applied to other expensive black-box tuning settings, such as hardware/compiler autotuning or simulation-based design, where cheaper approximations exist.
  • A testable prediction follows from the 'less can be more' thesis: on any system where a cheap setting ranks configurations nearly identically to the target, MFTune-style seeding should beat direct tuning, and the benefit should grow with the cost gap between the two settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a fidelity-aware configuration tuning framework, MFTune, which proactively searches over a multi-dimensional space of imperfect-fidelity settings, selects a 'fair' setting that balances measurement cost and Spearman-rank-based fidelity perfection, performs diversity-preserving tuning under that setting to generate seeds, and then uses those seeds to warm-start a perfect-fidelity GA. The work is evaluated on six real-world configurable systems against ten baselines, with claims of best Scott-Knott rank on 5/6 systems, up to 19.34% performance improvement, and substantial budget savings. The paper also defines a taxonomy of fidelity factors and provides an exploratory study of PostgreSQL to motivate the non-monotonic cost-perfection relationship.

Significance. If the claims hold, the paper makes a useful conceptual contribution: it formalizes multi-fidelity tuning for configurable systems in a way that does not assume a monotonic cost-perfection relationship and that allows multiple fidelity factors to interact. The reported experimental effort is substantial—six systems, 19 months of CPU time, ten diverse baselines, statistical testing, and an ablation study—and the code/data are promised publicly, which strengthens reproducibility. The key ideas of proactive fidelity discovery and seed-archive transfer between fidelity settings are plausible and well motivated. However, two load-bearing methodological issues (small-sample fidelity estimation and on-test-set parameter selection) currently leave the quantitative claims insufficiently supported for a definitive acceptance.

major comments (4)
  1. [§4.1, Eq. (4), Algorithm 2, §5.5] Fidelity perfection is estimated as the Spearman correlation between only l=10 LHS-sampled configurations (Algorithm 2, Eq. (4); §5.5 sets l=10). With n=10, the sampling distribution of Spearman's rho is very wide—the standard error is about 0.33 under the null and confidence intervals for rho near 0.8 still span several tenths—while the differences among candidate fidelity settings shown in Figure 10 are often only about 0.1. NSGA-II in Algorithm 2 is therefore optimizing a quantity that may be dominated by sampling noise. This is load-bearing because RQ3 shows that removing the discovery phase (MFTune-I) degrades results, meaning the quality of z_fair directly affects the main claims. The paper acknowledges the l trade-off in §7.3, but no sensitivity analysis or validation on larger held-out samples is provided. I request either a sensitivity study for l (e.g., l=10,20,50), or a verifi
  2. [§5.5, §6.4] The parameter alpha=0.5 is selected from a sensitivity analysis conducted on the same six systems used in the main evaluation (Figure 9, RQ4). The main results in Table 4 then use this tuned value, while baselines use default or literature-reported settings. This creates an information-leakage/overfitting risk: the reported 83.33% best-rank result may partly reflect fitting alpha to these specific benchmarks. I recommend a nested or held-out procedure (e.g., choose alpha on a subset of systems and evaluate on the rest, or report the main comparison for several alpha values), or a clear argument that alpha is not task-specific.
  3. [Table 4, §5.2] The comparison with multi-fidelity baselines (Hyperband, BOHB, DEHB, PriorBand) adapts them by 'choosing the factor that is the most influential on cost,' but the manuscript does not state how this factor is identified or whether the chosen factor is favorable for those algorithms. Since these baselines are claimed to be outperformed, the fairness of this adaptation is important. Please specify the factor choices per system and, ideally, report sensitivity to at least an alternative factor selection.
  4. [Table 4] On Tomcat and Httpd, MFTune's standard deviations are very large relative to the mean differences (e.g., Tomcat: MFTune 3323.13 ± 843.83 vs. FLASH+ 3309.26 ± 730.02). Scott-Knott ESD may still rank MFTune first, but the practical significance of a 0.4% mean improvement with overlapping distributions is unclear. Please report effect sizes, confidence intervals, or the number of runs where MFTune beats each baseline, so the 'considerably better' claim can be assessed beyond mean ranks.
minor comments (5)
  1. [§7.3] The text says 'sensitivity analysis (RQ3)' but the alpha sensitivity study is RQ4; this cross-reference should be corrected.
  2. [Table 3 vs. reference [58]] FLASH+ is listed as 2018 in Table 3, but reference [58] is dated 2020. Please reconcile the year.
  3. [Algorithm 3, line 1] The expression 'B/4×τ_z_fair' is ambiguous: it could mean B/(4τ) or (B/4)×τ. Clarify the intended arithmetic.
  4. [Figure 10] The three panels lack axis titles and a shared legend for the star/circle markers; adding these would make the evolution of the Pareto front easier to follow.
  5. [§5.5] The choice of population sizes m=10 and n=20, and the equal quarter-budget split, are given without justification or sensitivity analysis. Even if these are secondary, a sentence explaining the rationale would help.

Circularity Check

0 steps flagged

No significant circularity: MFTune's fidelity-selection and seeding chain is not equivalent to its inputs; final evaluations are independent measurements under the perfect-fidelity setting.

full rationale

The paper's derivation chain is empirical rather than definitional. MFTune selects a fair imperfect-fidelity setting by maximizing rho/tau, where rho is the Spearman correlation between performance ranks on l=10 LHS configurations (Eq. 4). This is an operational definition of "fair", not a fitted value of the final tuning result; the eventual best configuration is actually measured under the perfect-fidelity setting z* (Algorithms 3-4). The same initial archive later seeds the perfect-fidelity tuning, but those 10 configurations are random LHS points and are not optimized by the fidelity-selection step; they enter the archive only as candidates, and the final ranking is decided by fresh measurements under z*. The paper's acknowledged limitations—small l for correlation estimation, and alpha chosen after sensitivity analysis—are robustness/validation concerns, not cases where the conclusion is equivalent to its inputs. Self-citations appear only for baseline tuners and standard parameter choices (e.g., [18], [19], [24]) and are not load-bearing for the central premise. No step in the paper's equations reduces a predicted quantity to a fitted input.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or formal entities. It does introduce a conceptual taxonomy of fidelity factors and the notion of a 'fair imperfect-fidelity setting', which are useful abstractions but not independent falsifiable entities. The main free parameters are heuristic choices (alpha, l, m, n, budget split) that affect the method's performance. The axioms are mostly domain assumptions about how fidelity should be measured and whether observations on one system generalize.

free parameters (5)
  • alpha (probability of archiving diverse configurations under imperfect fidelity) = 0.5
    Selected via sensitivity analysis over {0.1,0.3,0.5,0.7,0.9} on the same six systems used for the main evaluation (Section 6.4). This is a form of fitting to the test data.
  • l (number of configurations for estimating fidelity perfection) = 10
    Chosen as a trade-off between estimation reliability and discovery cost (Section 5.5, Section 7.3). Not derived from theory.
  • m (fidelity population size in NSGA-II) = 10
    Set smaller than n because measuring fidelity settings is more expensive (Section 5.5).
  • n (configuration population size in GA) = 20
    Adopted from prior work to ensure fair comparison (Section 5.5).
  • Budget split (quarter per phase) = B/4 each
    A design choice stated in Section 4: equal split among discovery, seeding, and perfect-fidelity assuring phases.
axioms (5)
  • domain assumption Spearman rank correlation is an appropriate measure of fidelity perfection between an imperfect and the perfect fidelity setting.
    Used in Equation 4 to quantify how well one environment approximates another. This assumes that ranking preservation captures tuning usefulness, which is not formally justified.
  • ad hoc to paper A small set of 10 LHS-sampled configurations is representative for estimating fidelity perfection across a high-dimensional configuration space.
    Section 4.1 uses l=10 configurations to compute the Spearman correlation. The paper does not test whether this sample size is sufficient.
  • domain assumption The non-monotonic relationship between cost and fidelity perfection observed on PostgreSQL generalizes to other configurable systems.
    Observation 2 is based on 10 imperfect-fidelity settings on PostgreSQL (Section 3.3). The paper assumes this holds for all six systems and uses it to motivate the design.
  • domain assumption Fidelity factors can be meaningfully classified into Budget-, Workload-, and Dataset-related categories.
    The taxonomy in Figure 2 is presented as a general lens for identifying fidelity factors. The paper does not prove completeness or mutual exclusivity.
  • domain assumption The perfect-fidelity setting is the one with highest measurement cost.
    In the experiments (Section 5.1, Table 2), the authors deliberately set the most expensive setting as the perfect-fidelity ground truth. This may not hold in all real deployment scenarios.

pith-pipeline@v1.3.0-alltime-deepseek · 24477 in / 10958 out tokens · 118525 ms · 2026-08-05T00:19:08.823891+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Less Is More: Tuning Configurable Systems with Imperfect Fidelity." pith.science (2026). https://pith.science/paper/CQ5ZABAA

@misc{pith2026260800759,
  author       = {Pith},
  title        = {Pith review of: Less Is More: Tuning Configurable Systems with Imperfect Fidelity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQ5ZABAA}},
  note         = {Machine review of arXiv:2608.00759}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Configuration tuning is essential for optimizing the performance of highly configurable systems, e.g., throughput or runtime, under a given environment. Yet, this is a challenging process as there can be many options to tune, and configuration measurement is often highly expensive. In this paper, we demonstrate the phenomenon of ``less can be more'': system configuration tuning can be greatly improved with much superior budget utilization by partially tuning under the imperfect-fidelity---an environment that is similar, but cheaper to measure, compared with the concerned perfect-fidelity of environment under which the system should be tuned. We codify a conceptual framework of fidelity for configurable systems, drawing on which allows us to propose MFTune, a tuner that proactively explores in the space of $>10^4$ possible imperfect-fidelity settings to approximate a useful one, which strikes for the wideness of tuning. This creates high-quality seeds for the perfect-fidelity, which in turn ensures the tuning depth. Experiment results against $10$ state-of-the-art tuners, obtained from running diverse real-world systems for $19$ months $24 \times 7$, show that MFTune performs considerably better on $83.33$\% cases with up to $19.34\%$ improvement while achieving hours of budget saving in general.

Figures

Figures reproduced from arXiv: 2608.00759 by Miqing Li, Tao Chen, Yulong Ye.

Figure 1
Figure 1. Figure 1: Tuning PostgreSQL under different environ￾ments/fidelity settings via a random search tuner. fidelity-A (time=180s; size=5000k) is the “target perfection”; fidelity-B (time=30s; size=5000k) and fidelity-C (time=60s; size=1050k) are two imperfect fidelity settings. “Tuning in fidelity-A test for fidelity-B” means that the tuning is guided by configu￾rations measured under fidelity-A and the best one found t… view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of fidelity factors. illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) and (b) visualize the landscapes (processed [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: and Algorithm 1 illustrate the workflow of MFTune. Given a perfect-fidelity setting, the key idea is to first explore the fidelity space to identify a “fair” imperfect-fidelity setting—one that of￾fers good fidelity perfection while incurring a much lower cost, hence improving budget utilization. We then conduct a specifi￾cally designed diversity-preserving tuning under the selected fair imperfect-fidelity… view at source ↗
Figure 5
Figure 5. Figure 5: Discovering the fair imperfect-fidelity setting. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Finding seeds under fair imperfect-fidelity setting. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Seeded tuning under perfect-fidelity setting. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Tuning trajectories with and without multi-fidelity [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The sensitivity of MFTune to 𝛼 values over all sys￾tems/runs (smaller normalized performance is preferred). 0.6 0.7 0.8 0.9 1 40 50 60 70 Fidelity perfection Cost (s) (a) Iteration 𝑘 + 1 0.6 0.7 0.8 0.9 1 40 50 60 70 Fidelity perfection Cost (s) (b) Iteration 𝑘 + 2 0.6 0.7 0.8 0.9 1 40 50 60 70 fair imperfect￾fidelity setting Fidelity perfection Cost (s) (c) Iteration 𝑘 + 3 [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 10
Figure 10. Figure 10: The changes within imperfect-fidelity discovering [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Tuning trajectory of MySQL under the fair imperfect-fidelity and testing the corresponding configura￾tions found thereof under the perfect-fidelity setting. 7.3 Threats to Validity Internal threats to validity could be raised from the parameter setting of 𝛼, which controls the probability of triggering full-fidelity evaluations during low-fidelity guided tuning. In our study, we set 𝛼 = 0.5, following emp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 55 canonical work pages

  1. [1]

    2024. Clang. https://releases.llvm.org/download.html. Version: 17.0.6. Accessed: 2024-02-20

    work page 2024

  2. [2]

    2024. Csmith. https://github.com/csmith-project/csmith. Accessed: 2024-02-20

    work page 2024

  3. [3]

    2024. Gcc. https://gcc.gnu.org/pub/gcc/releases/gcc-14.2.0/. Version: 14.2. Ac- cessed: 2024-02-20

    work page 2024

  4. [4]

    2024. Httpd. https://httpd.apache.org/download.cgi. Version: 2.4.57. Accessed: 2024-02-20

    work page 2024

  5. [5]

    2024. MySQL. https://dev.mysql.com/doc/relnotes/mysql/5.7/en/. Version: 5.7.19. Accessed: 2024-02-20

    work page 2024

  6. [6]

    PostgreSQL

    2024. PostgreSQL. https://www.postgresql.org/docs/release/12.7/. Version: 12.7. Accessed: 2024-02-20

    work page 2024

  7. [7]

    Sysbench

    2024. Sysbench. https://github.com/akopytov/sysbench. Accessed: 2024-02-20

    work page 2024

  8. [8]

    2024. Tomcat. https://tomcat.apache.org/download-10.cgi. Version: 10.1.34. Accessed: 2024-02-20

    work page 2024

  9. [9]

    2024. Wrk. https://github.com/wg/wrk. Accessed: 2024-02-20

    work page 2024

  10. [10]

    Gordon, and Bohan Zhang

    Dana Van Aken, Andrew Pavlo, Geoffrey J. Gordon, and Bohan Zhang. 2017. Automatic Database Management System Tuning Through Large-scale Machine Learning. InProceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Conference 2017, Chicago, IL, USA, May 14-19, 2017, Semih Salihoglu, Wenchao Zhou, Rada Chirkova, Jun Yang, and Dan Su...

    arXiv 2017

  11. [11]

    Awad, Neeratyoy Mallik, and Frank Hutter

    Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. 2021. DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, Zhi-Hua Zhou (Ed.). ijcai.org, 2147–2153. doi:10.249...

    doi:10.24963/ijcai.2021/296 2021

  12. [12]

    Thomas Bäck and Hans-Paul Schwefel. 1993. An Overview of Evolutionary Algorithms for Parameter Optimization.Evol. Comput.1, 1 (1993), 1–23. doi:10. 1162/EVCO.1993.1.1.1

    work page 1993

  13. [13]

    Aydt, Quincey Koziol, and Marc Snir

    Babak Behzad, Huong Vu Thanh Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth A. Aydt, Quincey Koziol, and Marc Snir. 2013. Taming parallel I/O complexity with auto-tuning. InInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC’13, Denver, CO, USA - November 17 - 21, 2013, William Gropp and Satoshi Matsuoka (Eds....

    arXiv 2013

  14. [14]

    Timur Carstensen, Neeratyoy Mallik, Frank Hutter, and Martin Rapp. 2025. Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization. InPro- ceedings of the Fourth International Conference on Automated Machine Learning (Proceedings of Machine Learning Research, Vol. 293), Leman Akoglu, Carola Doerr, Jan N. van Rijn, Roman Garnett, and Jacob ...

    work page 2025

  15. [15]

    Sampling

    Jianfeng Chen, Vivek Nair, Rahul Krishna, and Tim Menzies. 2019. "Sampling" as a Baseline Optimizer for Search-Based Software Engineering.IEEE Trans. Software Eng.45, 6 (2019), 597–614. doi:10.1109/TSE.2018.2790925

    work page arXiv 2019

  16. [16]

    Junjie Chen, Ningxin Xu, Peiqi Chen, and Hongyu Zhang. 2021. Efficient Compiler Autotuning via Bayesian Optimization. In43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 1198–

    work page 2021

  17. [17]

    Pengzhou Chen and Tao Chen. 2026. CDS4RAG: Cyclic Dual-Sequential Hyper- parameter Optimization for RAG. InProceedings of the 35th International Joint Conference on Artificial Intelligence, IJCAI 2026, Bremen, Germany, 15-21 August

    work page 2026

  18. [18]

    Pengzhou Chen and Tao Chen. 2026. PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning. In2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE)

    work page 2026

  19. [19]

    Pengzhou Chen, Tao Chen, and Miqing Li. 2024. MMO: Meta Multi- Objectivization for Software Configuration Tuning.IEEE Trans. Software Eng.50, 6 (2024), 1478–1504. doi:10.1109/TSE.2024.3388910

    work page arXiv 2024

  20. [20]

    Pengzhou Chen, Jingzhi Gong, and Tao Chen. 2025. Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning.IEEE Trans. Software Eng. 51, 2 (2025), 548–580. doi:10.1109/TSE.2025.3525955

    work page arXiv 2025

  21. [21]

    Tao Chen and Rami Bahsoon. 2015. Toward a Smarter Cloud: Self-Aware Au- toscaling of Cloud Configurations and Resources.Computer48, 9 (2015), 93–96. doi:10.1109/MC.2015.278

    work page doi:10.1109/mc.2015.278 2015

  22. [22]

    Tao Chen, Rami Bahsoon, and Xin Yao. 2018. A Survey and Taxonomy of Self- Aware and Self-Adaptive Cloud Autoscaling Systems.ACM Comput. Surv.51, 3 (2018), 61:1–61:40. doi:10.1145/3190507

    doi:10.1145/3190507 2018

  23. [23]

    Tao Chen, Ke Li, Rami Bahsoon, and Xin Yao. 2018. FEMOSAA: Feature-guided and knee-driven multi-objective optimization for self-adaptive software.ACM Transactions on Software Engineering and Methodology (TOSEM)27, 2 (2018), 1–50

    work page 2018

  24. [24]

    Tao Chen and Miqing Li. 2021. Multi-objectivizing software configuration tun- ing. InESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, Au- gust 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 453–465. doi:...

    work page arXiv 2021

  25. [25]

    Tao Chen and Miqing Li. 2024. Adapting Multi-objectivized Software Configura- tion Tuning.Proc. ACM Softw. Eng.1, FSE (2024), 539–561. doi:10.1145/3643751

    work page doi:10.1145/3643751 2024

  26. [26]

    Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, Octo...

    work page 2018

  27. [27]

    Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. 2022. Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation.Journal of Artificial Intelligence Research74 (2022), 1269–1349

    work page 2022

  28. [28]

    2000.Multidimensional scaling

    Trevor F Cox and Michael AA Cox. 2000.Multidimensional scaling. CRC press

    work page 2000

  29. [29]

    K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist mul- tiobjective genetic algorithm: NSGA-II.IEEE Trans. Evol. Comput.6, 2 (2002), 182–197. doi:10.1109/4235.996017

    arXiv 2002

  30. [30]

    Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Ex- trapolation of Learning Curves. InProceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, Qiang Yang and Michael J. Wool...

    work page 2015

  31. [31]

    Judith Echevarrieta, Etor Arza, and Aritz Pérez. 2025. Speeding-Up Evolutionary Algorithms to Solve Black-Box Optimization Problems.IEEE Trans. Evol. Comput. 29, 1 (2025), 117–131. doi:10.1109/TEVC.2024.3352450

    work page arXiv 2025

  32. [32]

    Awad, Marius Lindauer, and Frank Hut- ter

    Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, and Frank Hut- ter. 2021. HPOBench: A Collection of Reproducible Multi-Fidelity Bench- mark Problems for HPO. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and B...

    work page 2021

  33. [33]

    Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and Ef- ficient Hyperparameter Optimization at Scale. InProceedings of the 35th In- ternational Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PML...

    work page 2018

  34. [34]

    Alexander IJ Forrester, András Sóbester, and Andy J Keane. 2007. Multi-fidelity optimization via surrogate modelling.Proceedings of the royal society a: mathe- matical, physical and engineering sciences463, 2088 (2007), 3251–3269

    work page 2007

  35. [35]

    Jingzhi Gong and Tao Chen. 2023. Predicting Software Performance with Divide- and-Learn. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, Satish Chandra, Kelly Blincoe, and Paolo Tonella (Eds.). ACM, 858–870. doi...

    arXiv 2023

  36. [36]

    Jingzhi Gong and Tao Chen. 2024. Predicting Configuration Performance in Multiple Environments with Sequential Meta-Learning.Proc. ACM Softw. Eng.1, FSE (2024), 359–382. doi:10.1145/3643743

    doi:10.1145/3643743 2024

  37. [37]

    Jingzhi Gong, Tao Chen, and Rami Bahsoon. 2025. Dividable Configuration Performance Learning.IEEE Trans. Software Eng.51, 1 (2025), 106–134. doi:10. 1109/TSE.2024.3491945

    work page arXiv 2025

  38. [38]

    Xue Han and Tingting Yu. 2016. An Empirical Study on Performance Bugs for Highly Configurable Software Systems. InProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8-9, 2016. ACM, 23:1–23:10. doi:10. 1145/2961111.2962602

    work page arXiv 2016

  39. [39]

    Jan Hauke and Tomasz Kossowski. 2011. Comparison of values of Pearson’s and Spearman’s correlation coefficients on the same sets of data.Quaestiones geographicae30, 2 (2011), 87–93

    2011

  40. [40]

    Haochen He, Zhouyang Jia, Shanshan Li, Yue Yu, Chenglong Zhou, Qing Liao, Ji Wang, and Xiangke Liao. 2022. Multi-Intention-Aware Configuration Selection for Performance Tuning. In44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ACM, 1431–1442. doi:10.1145/3510003.3510094

    work page arXiv 2022

  41. [41]

    An Empirical Comparison of Model Validation Techniques for Defect Prediction Models

    Steffen Herbold. 2017. Comments on ScottKnottESD in Response to "An Empirical Comparison of Model Validation Techniques for Defect Prediction Models".IEEE Trans. Software Eng.43, 11 (2017), 1091–1094. doi:10.1109/TSE.2017.2748129

    work page arXiv 2017

  42. [42]

    Abir Hossen, Mohammad Ali Javidian, Vignesh Narayanan, Jason M

    Md. Abir Hossen, Mohammad Ali Javidian, Vignesh Narayanan, Jason M. O’Kane, and Pooyan Jamshidi. 2026. Multi-Objective Multi-Fidelity Bayesian Optimization with Causal Priors.CoRRabs/2602.00788 (2026). arXiv:2602.00788 doi:10.48550/ ARXIV.2602.00788

    work page doi:10.48550/arxiv.2602.00788 2026

  43. [43]

    Qinghao Hu, Zhisheng Ye, Meng Zhang, Qiaoling Chen, Peng Sun, Yonggang Wen, and Tianwei Zhang. 2023. Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters. In17th USENIX Symposium on Operating Sys- tems Design and Implementation, OSDI 2023, Boston, MA, USA, July 10-12, 2023, Roxana Geambasu and Ed Nightingale (Eds.). USENIX Association, 757–...

    work page 2023

  44. [44]

    Yi-Qi Hu, Yang Yu, Wei-Wei Tu, Qiang Yang, Yuqiang Chen, and Wenyuan Dai

    work page

  45. [45]

    Hoos, and Kevin Leyton-Brown

    Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential Model- Based Optimization for General Algorithm Configuration. InLearning and Intel- ligent Optimization - 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers (Lecture Notes in Computer Science, Vol. 6683), Carlos A. Coello Coello (Ed.). Springer, 50...

    doi:10.1007/978-3-642-25566-3_40 2011

  46. [46]

    Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic Best Arm Identi- fication and Hyperparameter Optimization. InProceedings of the 19th Interna- tional Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 (JMLR Workshop and Conference Proceedings, Vol. 51), Arthur Gretton and Christian C. Robert (Eds.). JM...

    work page 2016

  47. [47]

    Pooyan Jamshidi and Giuliano Casale. 2016. An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems. In24th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommuni- cation Systems, MASCOTS 2016, London, United Kingdom, September 19-21, 2016. IEEE Computer Society, 39–48. doi:10.1109/MASCOTS.2016.17

    work page doi:10.1109/mascots.2016.17 2016

  48. [48]

    Schneider, and Barnabás Póc- zos

    Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider, and Barnabás Póc- zos. 2017. Multi-fidelity Bayesian Optimisation with Continuous Approximations. InProceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee W...

    work page 2017

  49. [49]

    Konstantinos Kanellis, Cong Ding, Brian Kroth, Andreas Müller, Carlo Curino, and Shivaram Venkataraman. 2022. LlamaTune: Sample-Efficient DBMS Configu- ration Tuning.Proc. VLDB Endow.15, 11 (2022), 2953–2965. doi:10.14778/3551793. 3551844

    doi:10.14778/3551793 2022

  50. [50]

    Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter

    work page

  51. [51]

    Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.J. Mach. Learn. Res.18 (2017), 185:1–185:52. https://jmlr.org/papers/ v18/16-558.html

    work page 2017

  52. [52]

    Yang Li, Yu Shen, Jiawei Jiang, Jinyang Gao, Ce Zhang, and Bin Cui. 2021. MFES- HB: Efficient Hyperband with Multi-Fidelity Quality Measurements. InThirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sym- posium on Educational Advances i...

    work page doi:10.1609/aaai.v35i10.17031 2021

  53. [55]

    Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, and Frank Hutter. 2023. PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans...

    work page 2023

  54. [56]

    Mary L McHugh. 2011. Multiple comparison analysis testing in ANOVA.Bio- chemia medica21, 3 (2011), 203–209

    work page 2011

  55. [57]

    Michael D. McKay. 1992. Latin Hypercube Sampling as a Tool in Uncertainty Analysis of Computer Models. InProceedings of the 24th Winter Simulation Con- ference, Arlington, V A, USA, December 13-16, 1992, Robert C. Crain (Ed.). ACM Press, 557–564. doi:10.1145/167293.167637

    work page arXiv 1992

  56. [58]

    Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel. 2020. Finding Faster Configurations Using FLASH.IEEE Trans. Software Eng.46, 7 (2020), 794–

    work page 2020

  57. [59]

    Arman Shahbazian, Suhrid Karthik, Yuriy Brun, and Nenad Medvidovic. 2020. eQual: informing early design decisions. InESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1039–10...

    doi:10.1145/3368089 2020

  58. [60]

    Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. InAdvances in Neural In- formation Processing Systems 25: 26th Annual Conference on Neural Infor- mation Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, Peter L. Bartlett, Fer...

    work page 2012

  59. [61]

    Zezhen Xiang, Jingzhi Gong, and Tao Chen. 2026. Dually Hierarchical Drift Adaptation for Online Configuration Performance Learning. In2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE)

    work page 2026

  60. [62]

    Gangda Xiong and Tao Chen. 2025. CoTune: Co-evolutionary Configuration Tuning. In40th IEEE/ACM International Conference on Automated Software Engi- neering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1490–1502. doi:10.1109/ASE63991.2025.00126

    arXiv 2025

  61. [63]

    Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015. Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. InProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2...

    arXiv 2015

  62. [64]

    Yulong Ye, Tao Chen, and Miqing Li. 2025. Distilled Lifelong Self-Adaptation for Configurable Systems. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1333–

    work page 2025

  63. [65]

    Yulong Ye, Hongyuan Liang, Chao Jiang, Miqing Li, and Tao Chen. 2026. Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapes.ACM Trans. Softw. Eng. Methodol.(March 2026). Just Accepted. doi:10.1145/3803859

    doi:10.1145/3803859 2026

  64. [66]

    Xinyi Zhang, Zhuo Chang, Yang Li, Hong Wu, Jian Tan, Feifei Li, and Bin Cui

    work page

  65. [67]

    Xinyi Zhang, Hong Wu, Zhuo Chang, Shuowei Jin, Jian Tan, Feifei Li, Tieying Zhang, and Bin Cui. 2021. ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases. InSIGMOD ’21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). A...

    work page arXiv 2021

  66. [68]

    Xinyi Zhang, Hong Wu, Yang Li, Zhengju Tang, Jian Tan, Feifei Li, and Bin Cui

  67. [69]

    Mingxuan Zhu and Dan Hao. 2023. Compiler Auto-Tuning via Critical Flag Selection. In38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 1000–1011. doi:10.1109/ASE56229.2023.00209

    arXiv 2023

  68. [70]

    Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, and Yingchun Yang. 2017. BestConfig: tapping the perfor- mance potential of systems via automatic configuration tuning. InProceedings of the 2017 Symposium on Cloud Computing, SoCC 2017, Santa Clara, CA, USA, September 24-27, 2017. ACM, 338–350. doi:10.1145/3127479....

    work page arXiv 2017

  69. [811]

    doi:10.1109/TSE.2018.2870895

    arXiv 2018

  70. [1209]

    doi:10.1109/ICSE43902.2021.00110

    arXiv 2021

  71. [1345]

    doi:10.1109/ICSE55347.2025.00094

    arXiv 2025

  72. [1821]

    doi:10.14778/3538598.3538604

    arXiv

  73. [2017]

    Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. InProceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA (Proceedings of Machine Learning Research, Vol. 54), Aarti Singh and Xiaojin (Jerry) Zhu (Eds.). PMLR, 528–536. http://proce...

    work page 2017

  74. [2019]

    Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion. InThe Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Ja...

    work page doi:10.1609/aaai.v33i01.33013846 2019

  75. [2022]

    VLDB Endow.15, 9 (2022), 1808–

    Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation.Proc. VLDB Endow.15, 9 (2022), 1808–

    work page 2022

  76. [2023]

    VLDB Endow.17, 3 (2023), 539–552

    An Efficient Transfer Learning Based Configuration Adviser for Database Tuning.Proc. VLDB Endow.17, 3 (2023), 539–552. doi:10.14778/3632093.3632114

    arXiv 2023