Pith. sign in

REVIEW 4 major objections 7 minor 113 references

Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that higher surrogate-model accuracy often fails to improve—and can actively degrade—configuration tuning quality, a claim tested on 13,612 cases across 10 models, 17 tuners, and 29 systems.

desk verdict A large, statistically careful empirical study that convincingly shows aggregate model accuracy is a poor predictor of tuning quality, though it stops short of refuting accuracy-as-a-whole because it never tests local accuracy. read the letter →

arxiv 2501.01876 v1 pith:6RDZMFMV submitted 2025-01-03 cs.SE cs.AI

Pengzhou Chen , Jingzhi Gong , Tao Chen This is my paper · ORCID
classification cs.SEcs.AI
keywords configurationtuningsurrogatemodelaccuracysoftwaremodel-basedoptimizationfitnesslandscapeanalysisperformancepredictionconfigurablesystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to test a widely held belief in software configuration tuning: that a more accurate surrogate model necessarily produces better tuning outcomes. Using an empirical corpus of 13,612 cases covering 10 models, 17 tuners, and 29 systems, it finds that higher accuracy yields no statistically significant improvement in tuning quality in up to 58% of cases under some settings, and actually degrades quality in up to 24%. The paper's central message is that aggregate prediction error, measured by MAPE or a rank-difference metric, is not a reliable proxy for a model's usefulness to a tuner, and that the community should stop treating accuracy as the dominant yardstick for model-based configuration tuning. A reader should care because model accuracy currently drives how configuration-performance models are published, compared, and criticized; if the claim holds, the evaluation criteria and research effort in this area need rethinking.

What carries the argument

The load-bearing apparatus has three parts. First, the measurement corpus: two accuracy metrics (mean absolute percentage error, $\text{MAPE}=\frac{1}{n}\sum_{i=1}^{n}\left|\frac{y_i-\hat{y}_i}{y_i}\right|\cdot 100\%$, and a sample-size-standardized rank difference $\frac{1}{n}\times \mu RD$), two tuning-quality metrics (the measured performance of the best returned configuration and an efficiency ratio of measurements needed to reach a baseline), and 13,612 cases built from 10 models, 17 tuners, and 29 systems with 30 repeated runs each. Second, the statistical apparatus—Mann–Whitney U-tests, Scott–Knott effect-size difference ranking, and Spearman's rank correlation $\rho$—that converts raw results into win/tie/lose counts and correlation strengths. Third, the interpretive machinery borrowed from fitness landscape analysis: fitness distance correlation $\varrho$ (the correlation between performance and distance to the nearest global optimum), correlation length $\ell$ (the autocorrelation of neighboring configurations' performances, measuring ruggedness), and the deviation between the model-predicted and true global optimum in both Hamming distance and performance. This third part carries the explanation: a tuner explores the landscape the model emulates, and models with similar error can inhabit very different landscapes (or vice versa), so error alone cannot predict the tuning outcome.

What would settle it

Recompute model accuracy on the subset of test configurations closest to the true optimum (for example within a small Hamming distance) or on the top-ranked predicted configurations: if this near-optimum accuracy correlates strongly and positively (Spearman $\rho > 0.7$) with the achieved tuning quality for a majority of the 29 systems, then the paper's conclusion would be shown to be an artifact of whole-space aggregate accuracy rather than a failure of accuracy as such.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that 'accuracy can lie': the correlation between a surrogate model's aggregate test-set accuracy and the quality of the configuration the tuner finds is weak, negligible, or even negative in the large majority of cases. Across the corpus, the most accurate model is the best for tuning quality in only 14–45% of cases depending on the metric combination, and the least accurate model marks the worst tuning outcomes in a comparably small share; in up to 97% of cases the accuracy–quality correlation is negligible or negative. The study also finds that the model originally packaged inside a sequential model-based tuner is sub-optimal in about 46% of cases, and that batch-style tuners whose pre-trained model is never updated lose to model-free tuning on performance in 28 of 29 systems. It further quantifies the minimum accuracy change needed to move tuning quality significantly, a threshold that depends on the accuracy range (a model at 30–40% MAPE needs roughly 13% improvement, while a model below 10% MAPE needs under 1%). The offered explanation is that tuning quality is governed by the configuration landscape the model emulates—its guidance toward the optimum, its ruggedness, and the location of its predicted optimum—none of which aggregate accuracy captures.

Load-bearing premise

The study defines 'model accuracy' as aggregate prediction error (MAPE or a rank-difference measure) computed on a random held-out test set; if the accuracy the community actually relies on when preferring a model is something different, such as accuracy near the optimum or the ability to rank the top candidates, then the conclusion that accuracy can lie holds for the aggregate measure and not necessarily for the accuracy that matters.

Editorial extensions

If this is right

  • Model-based configuration tuning should stop using aggregate accuracy as the primary criterion for judging surrogate models, and evaluation should add landscape-based proxies such as guidance toward the optimum, ruggedness, and predicted-optimum deviation.
  • Claims of a percentage accuracy improvement should be benchmarked against the minimum meaningful accuracy change documented here, for example roughly 13% for a model in the 30–40% MAPE range, before being presented as useful for tuning.
  • Because the originally chosen model is sub-optimal in about 46% of cases and the best model only becomes clear during tuning, automated joint selection of the model–tuner pair is a natural bi-level optimization problem for future tuners.
  • Batch model-based tuners that pair a fixed pre-trained model with a model-free search should not be assumed to retain tuning quality: under a fair budget they lose to model-free tuners on performance in 28 of 29 systems, whereas sequentially updated models beat the model-free baseline in 16 of 29 systems on performance and up to 21 of 29 on efficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the accuracy–quality gap stems from the optimizer exploiting the emulated landscape rather than the error surface, then the same pattern should appear in other surrogate-guided search settings, such as hyperparameter optimization or simulation-based design; the paper does not make this generalization, but its mechanism implies it.
  • A testable extension would recompute accuracy on the test configurations the tuner actually visited, or on those nearest to the predicted optimum; if that 'local accuracy' correlates strongly with tuning quality, the right measurement is not error in general but error where the optimizer operates.
  • The landscape metrics used for post-hoc explanation could plausibly become a pre-deployment model-selection rule: pick the model whose emulated landscape best matches the measured landscape in fitness-distance correlation and correlation length before spending any tuning budget.
  • The paper's thresholds for minimum meaningful accuracy change could serve as a community benchmark, making a newly reported accuracy gain interpretable only in comparison with these figures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper reports a large-scale empirical study of how surrogate-model accuracy affects configuration tuning quality. It covers 10 models, 17 tuners, 29 systems, and four metrics, leading to 13,612 investigation cases, each repeated 30 times. The study answers five research questions: whether models help tuning quality (RQ1), whether originally chosen models are best for tuning (RQ2), whether the most/least accurate model yields the best/worst tuning quality (RQ3), how accuracy and tuning quality correlate (RQ4), and how much accuracy change is needed for meaningful tuning improvement (RQ5). The central claim is that the community's "accuracy is all" belief is misleading: higher aggregate model accuracy frequently does not improve tuning quality (up to 58% of cases) and can even degrade it (up to 24% of cases). The paper also proposes fitness-landscape metrics as an alternative lens for assessing model usefulness. The manuscript is careful statistically, using Mann-Whitney U-tests, Scott-Knott ESD, Spearman correlation, and a test-size validation on the five largest systems, and it provides a public repository for code and data.

Significance. If the central claim holds, this is an important result for model-based configuration tuning: it challenges the widespread practice of judging surrogate models mainly by aggregate prediction error, and it provides concrete evidence that the connection between model accuracy and tuning quality is weak and sometimes negative. The study's scale (29 systems, 10 models, 17 tuners) and its 30-run repetition are genuinely strong, and the paper is transparent about statistical procedures and open-science artifacts. The landscape-analysis viewpoint is a valuable new interpretation even if it is only presented on selected examples. However, the significance is tempered by a gap between the belief as stated in the motivating quotations and the accuracy metrics actually measured: the paper operationalizes accuracy as MAPE and 1/n x muRD on randomly held-out test sets, while the community quotations the paper itself cites concern accuracy near optima or on "best points." The empirical results therefore establish a claim about aggregate test-set accuracy rather than about the local or decision-oriented accuracy that may be what practitioners actually mean when they prefer "more accurate" models.

major comments (4)
  1. [Sections 2.4 and 3.5.1-3.5.2] The paper defines the "accuracy is all" belief in broad terms but measures accuracy exclusively as MAPE and 1/n x muRD on randomly held-out test sets. The motivating quotations, however, refer to accuracy in the regions that matter for tuning: Zhu et al. [108] complain that "Gaussian Process's predictions about best points are hardly accurate," and Wang et al. [92] claim the model can "accurately fit the target." These are claims about local or top-ranked accuracy, not about average error over a random sample of the configuration space. If practitioners care about accuracy near the optimum or about the ranking of top candidates, the paper's RQ3-RQ5 results show only that aggregate accuracy correlates weakly with tuning quality; they do not refute a belief about locally relevant accuracy. Please either add accuracy metrics evaluated on tuning-relevant configurations (e.g., error on top-k predicted configurations or near-optimum configurations) and re-run the analyses, or explicitly narrow the central claim to "aggregate accuracy can lie."
  2. [Sections 4.3-4.4 and abstract] The comparison in RQ3 and RQ4 is across ten different surrogate models, so model accuracy is confounded with model family, uncertainty-estimation mechanism, and interaction with the acquisition function. The causal wording in the abstract ("higher accuracy actually leads to no improvement" and "accuracy can lie") is not supported by Spearman correlations computed across different models, because a difference in accuracy between, say, a decision tree and a deep network is inseparable from many other differences between the models. I recommend either rephrasing the findings as associational/predictive claims about the usefulness of aggregate accuracy as a proxy, or adding a controlled analysis in which accuracy is varied within a fixed model family (e.g., through regularization strength, training-set size, or label noise) while the model class and the tuner remain unchanged.
  3. [Section 4.5.1 and Figures 9-10] The "minimum accuracy change" in RQ5 is estimated by selecting, for each anchor model, the smallest accuracy difference to any significantly better model, using many pairwise Mann-Whitney tests without multiple-comparison correction. The anchors are then binned by their own accuracy, and the reported mean and standard deviation of Delta are aggregated over all tuner-system combinations. The interpretation of Finding 6 depends on the number of samples per bin and on how often no significantly better model exists in a bin; please report per-bin sample sizes and the false-positive risk from the multiple comparisons, or the reported ranges such as [0.6, 85.9] may be taken to be more precise than the procedure warrants.
  4. [Section 5 (especially 5.3.2 and 5.3.3)] The landscape explanation is based on a small number of hand-picked examples (TOMCAT, LLVM, HSMGP, HSQLDB, SQLITE). The text explicitly says these are representative examples, but the broader conclusion that landscape properties explain why accuracy changes do not translate into tuning-quality changes would be substantially strengthened if the landscape metrics (FDC, correlation length, global-optimum deviation) were computed systematically across the full set of system-tuner-model combinations and compared against the RQ3-RQ5 outcomes. As presented, the landscape analysis is an exploratory interpretation rather than a tested explanation. Please add such a systematic analysis or soften the contribution claim to "observational case studies."
minor comments (7)
  1. [Section 3.5.1] The text refers to "MPAE" in the paragraph on MAPE; this appears to be a typo for MAPE.
  2. [Section 3.4.1] The sentence ends "prevnting the samples from being concentrated in certain parts"; please correct the typo "prevnting."
  3. [Section 3.5.3] The efficiency metric r = m/b is undefined when a tuner-model pair never reaches the baseline performance T within the budget; please specify how such cases are handled, because they occur in several reported results.
  4. [Section 4.5.2 / Figure 10 caption] The caption states "0.0 +/- 0.0 implies no samples can significantly improve the tuning for a range," but a zero mean with zero standard deviation could also occur if exactly one sample has Delta = 0; please clarify whether the entry means "no samples" or "a zero Delta."
  5. [Section 3.6.3] The statement that rho = 0 means the two metrics "do not correlate with each other in any way" is too strong for Spearman's rank correlation, which only detects monotonic association; please rephrase.
  6. [Table 1] The "Scale" column uses "Small" and "Large" to classify systems by search space; the paper's introduction of these terms in Section 3.2.1 should be referenced at the table caption for clarity.
  7. [Section 3.7] The computation of the 13,612 total is described in a long parenthetical equation; a small table or bullet list showing how the 5,220 accuracy cases and 8,392 tuning-quality cases are constructed would make the case count easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claims are empirical summaries with accuracy and tuning quality measured independently.

full rationale

The paper's central claim is that higher surrogate-model accuracy, as measured by MAPE and 1/n x muRD, frequently does not improve tuning quality, and can even degrade it. These two quantities are measured independently: accuracy comes from prediction error on held-out test samples (Sections 3.5.1 and 3.5.2), while tuning quality comes from measured performance and convergence efficiency of tuner-model pairs (Section 3.5.3). The findings in RQ3-RQ5 are descriptive counts, correlations, and minimal observed accuracy differences; none of these quantities is fitted to a target conclusion or defined in terms of the conclusion. The RQ5 delta values are summaries of observed pairwise comparisons, not predictions forced by construction. Self-citations to the authors' own models (e.g., DaL, references [34] and [37]) are used only as part of the model set and related work, and the main conclusions do not depend on any single model's performance. The landscape analysis in Section 5 is an interpretive device, not a derivation. The weakest point is construct validity: aggregate test-set MAPE may not capture the 'accuracy near the optimum' that some cited authors emphasize, but that is a measurement-target mismatch, not circular reasoning. Therefore the paper is not circular, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or computational entities. Its main load-bearing choices are the operationalization of accuracy, the reuse of prior datasets, and the selection of models, tuners, and statistical thresholds; all are documented in Sections 3.5 and 3.6.

free parameters (5)
  • hot_start_size = 20
    Set as the most common value found in prior sequential tuners (Section 3.4.1). It affects every sequential tuner's initial model and remaining budget.
  • convergence_budget_criterion = 100 consecutive measurements
    Used in pilot experiments to set each system's tuning budget (Section 3.4.3). Changing this criterion would change the budgets and hence the win/loss counts.
  • test_sample_size = min(10% of remaining measured configs, 1000), or all if fewer
    Testing data for accuracy metrics (Section 3.5.2). The choice is validated only on the five largest systems, and other systems rely on the same rule.
  • RQ5_accuracy_bins = MAPE: 11 ranges from 0-10 to >100; muRD: 4 ranges from 0-0.1 to >0.3
    Binning used to compute the required accuracy-change thresholds (Section 4.5). The resulting deltas shift with bin boundaries.
  • efficiency_baseline = worst tuner in each comparison set
    The efficiency metric r is defined relative to the worst tuner (Equation 4, Section 3.5.3), making reported efficiency improvements sensitive to which tuners are included.
assumptions (4)
  • domain assumption Model accuracy can be measured on a held-out random test set, and MAPE and muRD are the accuracy notions behind the 'accuracy is all' belief.
    Section 2.4 frames the belief using MAPE examples, and Section 3.5.1 defines accuracy as MAPE and muRD. If the community's accuracy claims concern local or ranking accuracy, the aggregate test-set measure may not capture them.
  • domain assumption Reusing measured datasets from prior work preserves valid performance values and noise characteristics.
    Section 3.2.2 states that option selections, workloads, and measurements follow prior work and that datasets are reused. No fresh measurement campaign verifies these values.
  • domain assumption The selected 10 models, 17 tuners, and 29 systems are representative of the field's practice.
    Sections 3.2 and 3.3 justify selection from key venues, but the findings are empirical generalizations from this sample and may not transfer to other systems or future tuners.
  • standard math Standard non-parametric statistical thresholds are appropriate for comparing tuning outcomes.
    Section 3.6 uses Mann-Whitney U, Scott-Knott ESD, and Spearman correlation with common significance levels. These are standard conventions, but still assumptions about what counts as a meaningful difference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning." pith.science (2026). https://pith.science/paper/6RDZMFMV

@misc{pith2026250101876,
  author       = {Pith},
  title        = {Pith review of: Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RDZMFMV}},
  note         = {Machine review of arXiv:2501.01876}
}
read the original abstract

To ease the expensive measurements during configuration tuning, it is natural to build a surrogate model as the replacement of the system, and thereby the configuration performance can be cheaply evaluated. Yet, a stereotype therein is that the higher the model accuracy, the better the tuning result would be. This "accuracy is all" belief drives our research community to build more and more accurate models and criticize a tuner for the inaccuracy of the model used. However, this practice raises some previously unaddressed questions, e.g., Do those somewhat small accuracy improvements reported in existing work really matter much to the tuners? What role does model accuracy play in the impact of tuning quality? To answer those related questions, we conduct one of the largest-scale empirical studies to date-running over the period of 13 months 24*7-that covers 10 models, 17 tuners, and 29 systems from the existing works while under four different commonly used metrics, leading to 13,612 cases of investigation. Surprisingly, our key findings reveal that the accuracy can lie: there are a considerable number of cases where higher accuracy actually leads to no improvement in the tuning outcomes (up to 58% cases under certain setting), or even worse, it can degrade the tuning quality (up to 24% cases under certain setting). We also discover that the chosen models in most proposed tuners are sub-optimal and that the required % of accuracy change to significantly improve tuning quality varies according to the range of model accuracy. Deriving from the fitness landscape analysis, we provide in-depth discussions of the rationale behind, offering several lessons learned as well as insights for future opportunities. Most importantly, this work poses a clear message to the community: we should take one step back from the natural "accuracy is all" belief for model-based configuration tuning.

Figures

Figures reproduced from arXiv: 2501.01876 by the authors.

Figure 1
Figure 1. Exampled accuracy (MAPE) and achieved tuning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrating the different types of tuners for configuration tuning in one tuning run. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparing µRD and its better standardized version under Decision Tree for system 7Z. TABLE 4: The p-values from the U-Test when comparing the MAPE (of all models) under our method of setting the testing sample size and that under using all remaining samples on the five largest systems over 30 runs. Model System DT RF LR SVR GP SPL DCT DeP HIP DaL STORM 1.00 0.85 0.95 0.54 0.90 0.97 0.74 0.85 0.97 0.95 SAC 0.81 0.63 … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The architecture and workflow of our empirical study. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Comparing the best tuner-model pair against the [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparing the original model in the sequential [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Counting the number of cases where the best (worst) [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Counting the number of cases on the nature/strength for the correlation between accuracy and tuning quality with [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The ∆ of model accuracy for creating significant improvements on tuning quality under sequential model-based tuners. +∆ and −∆ denote the improvements in tuning quality are the results of enhanced and worsened accuracy, respectively. The abbreviations are the same as …
Figure 10
Figure 10. Figure 10: The ∆ of model accuracy for creating significant improvements on tuning quality under batch model-based tuners. 0.0±0.0 implies no samples can significantly improve the tuning for a range. The formats are the same as [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Configuration landscapes derived from the model [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Configuration landscapes derived from the models [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Configuration landscapes derived from the final [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Configuration landscapes derived from the models [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Configuration landscapes derived from the models [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

113 extracted references · 44 canonical work pages

  1. [108]

    Bestconfig: tapping the performance potential of systems via automatic configuration tuning,

    Y. Zhu, J. Liu, M. Guo, Y. Bao, W. Ma, Z. Liu, K. Song, and Y. Yang, “Bestconfig: tapping the performance potential of systems via automatic configuration tuning,” in Proceedings of the 2017 Sym- posium on Cloud Computing, 2017, pp. 338–350

    work page 2017

  2. [92]

    Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,

    L. Wang, L. Yang, Y. Yu, W. Wang, B. Li, X. Sun, J. He, and L. Zhang, “Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,” in SoCC ’21: ACM Symposium on Cloud Computing, Seattle, WA, USA, November 1 - 4, 2021 , C. Curino, G. Koutrika, and R. Netravali, Eds. ACM, 2021, pp. 639–653. [Online]. Available: https://doi.org/10.1145...

    arXiv 2021

  3. [1]

    Mysqltuner,

    “Mysqltuner,” https://github.com/major/MySQLTuner-perl, retrieved on Sep 1, 2024

    2024

  4. [2]

    Model-based genetic algorithms for algorithm configuration,

    C. Ansótegui, Y. Malitsky, H. Samulowitz, M. Sellmann, and K. Tierney, “Model-based genetic algorithms for algorithm configuration,” in Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015 , Q. Yang and M. J. Wooldridge, Eds. AAAI Press, 2015, pp. 733–739. [Online...

    2015

  5. [3]

    A gender-based genetic algorithm for the automatic configuration of algorithms,

    C. Ansótegui, M. Sellmann, and K. Tierney, “A gender-based genetic algorithm for the automatic configuration of algorithms,” in International Conference on Principles and Practice of Constraint Programming. Springer, 2009, pp. 142–157

    2009

  6. [4]

    ACTGAN: automatic configuration tuning for software systems with generative adversarial networks,

    L. Bao, X. Liu, F. Wang, and B. Fang, “ACTGAN: automatic configuration tuning for software systems with generative adversarial networks,” in 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019 . IEEE, 2019, pp. 465–476. [Online]. Available: https://doi.org/10.1109/ASE.2019.00051

    arXiv 2019

  7. [5]

    Autoconfig: automatic configuration tuning for distributed message systems,

    L. Bao, X. Liu, Z. Xu, and B. Fang, “Autoconfig: automatic configuration tuning for distributed message systems,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 , M. Huchard, C. Kästner, and G. Fraser, Eds. ACM, 2018, pp. 29–40. [Online]. Available: https:/...

    arXiv 2018

  8. [6]

    Cm-casl: Comparison-based performance modeling of software systems via collaborative active and semisupervised learning,

    R. Cao, L. Bao, C. Wu, P . Zhangsun, Y. Li, and Z. Zhang, “Cm-casl: Comparison-based performance modeling of software systems via collaborative active and semisupervised learning,” Journal of Systems and Software, vol. 201, p. 111686, 2023

    2023

Show all 113 references
  1. [7]

    CM- CASL: comparison-based performance modeling of software systems via collaborative active and semisupervised learning,

    R. Cao, L. Bao, C. Q. Wu, P . Zhangsun, Y. Li, and Z. Zhang, “CM- CASL: comparison-based performance modeling of software systems via collaborative active and semisupervised learning,” J. Syst. Softw. , vol. 201, p. 111686, 2023. [Online]. Available: https://doi.org/10.1016/j....

    2023

  2. [8]

    Ptssbench: a performance evaluation platform in support of automated parameter tuning of software systems,

    R. Cao, L. Bao, P . Zhangsun, C. Q. Wu, S. Wei, R. Sun, R. Li, and Z. Zhang, “Ptssbench: a performance evaluation platform in support of automated parameter tuning of software systems,” Autom. Softw. Eng. , vol. 31, no. 1, p. 4, 2024. [Online]. Available: https://doi.org/10.10...

    2024 doi

  3. [9]

    Cgptuner: a contextual gaussian process bandit approach for the automatic tuning of IT configurations under varying workload conditions,

    S. Cereda, S. Valladares, P . Cremonesi, and S. Doni, “Cgptuner: a contextual gaussian process bandit approach for the automatic tuning of IT configurations under varying workload conditions,” Proc. VLDB Endow. , vol. 14, no. 8, pp. 1401–1413,

  4. [10]

    “sampling

    J. Chen, V . Nair, R. Krishna, and T. Menzies, ““sampling” as a baseline optimizer for search-based software engineering,” IEEE Transactions on Software Engineering , vol. 45, no. 6, pp. 597–614, 2018

    2018

  5. [11]

    Efficient compiler autotuning via bayesian optimization,

    J. Chen, N. Xu, P . Chen, and H. Zhang, “Efficient compiler autotuning via bayesian optimization,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 1198–1209

    2021

  6. [12]

    MMO: meta multi-objectivization for software configuration tuning,

    P . Chen, T. Chen, and M. Li, “MMO: meta multi-objectivization for software configuration tuning,” IEEE Trans. Software Eng. , vol. 50, no. 6, pp. 1478–1504, 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3388910

    2024

  7. [13]

    All versus one: an empirical comparison on retrained and incremental machine learning for modeling performance of adaptable software,

    T. Chen, “All versus one: an empirical comparison on retrained and incremental machine learning for modeling performance of adaptable software,” in Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS@ICSE, 2019,...

    2019

  8. [14]

    Lifelong dynamic optimization for self-adaptive systems: Fact or fiction?

    ——, “Lifelong dynamic optimization for self-adaptive systems: Fact or fiction?” in IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022 . IEEE, 2022, pp. 78–89. [Online]. Available: https://doi.org/10...

    2022

  9. [15]

    Planning landscape analysis for self-adaptive systems,

    ——, “Planning landscape analysis for self-adaptive systems,” in International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2022, Pittsburgh, P A, USA, May 22-24, 2022 , B. R. Schmerl, M. Maggio, and J. Cámara, Eds. ACM/IEEE, 2022, pp. 84–90. ...

    2022

  10. [16]

    Self-adaptive and online qos modeling for cloud-based software services,

    T. Chen and R. Bahsoon, “Self-adaptive and online qos modeling for cloud-based software services,” IEEE Trans. Software Eng. , vol. 43, no. 5, pp. 453–475, 2017. [Online]. Available: https://doi.org/10.1109/TSE.2016.2608826

    2017

  11. [17]

    Self-adaptive trade-off decision making for autoscaling cloud-based services,

    ——, “Self-adaptive trade-off decision making for autoscaling cloud-based services,” IEEE Trans. Serv. Comput. , vol. 10, no. 4, pp. 618–632, 2017. [Online]. Available: https://doi.org/10.1109/ TSC.2015.2499770

    2017

  12. [18]

    FEMOSAA: feature-guided and knee-driven multi-objective optimization for self-adaptive software,

    T. Chen, K. Li, R. Bahsoon, and X. Yao, “FEMOSAA: feature-guided and knee-driven multi-objective optimization for self-adaptive software,” ACM Trans. Softw. Eng. Methodol. , vol. 27, no. 2, pp. 5:1–5:50, 2018. [Online]. Available: https://doi.org/10.1145/3204459

    2018 doi

  13. [19]

    Multi-objectivizing software configuration tuning,

    T. Chen and M. Li, “Multi-objectivizing software configuration tuning,” in ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021 , D. Spinellis, G. Gousios, M. Chechik,...

    2021

  14. [20]

    Do performance aspirations matter for guiding software configuration tuning? an empirical investigation under dual performance objectives,

    ——, “Do performance aspirations matter for guiding software configuration tuning? an empirical investigation under dual performance objectives,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 3, pp. 68:1–68:41, 2023. [Online]. Available: https://doi.org/10.1145/3571853

    2023 doi

  15. [21]

    The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering,

    ——, “The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 1, pp. 5:1–5:40, 2023. [Online]. Available: https://doi.org/10.1145/ 3514233

    2023

  16. [22]

    Adapting multi-objectivized software configuration tuning,

    ——, “Adapting multi-objectivized software configuration tuning,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, pp. 539–561,

  17. [23]

    Tuning configuration of apache spark on public clouds by combining multi- objective optimization and performance prediction model,

    G. Cheng, S. Ying, and B. Wang, “Tuning configuration of apache spark on public clouds by combining multi- objective optimization and performance prediction model,” J. Syst. Softw. , vol. 180, p. 111028, 2021. [Online]. Available: https://doi.org/10.1016/j.jss.2021.111028

    2021

  18. [24]

    Hinnperf: Hierarchical interac- tion neural network for performance prediction of configurable systems,

    J. Cheng, C. Gao, and Z. Zheng, “Hinnperf: Hierarchical interac- tion neural network for performance prediction of configurable systems,” ACM Transactions on Software Engineering and Methodol- ogy, vol. 32, no. 2, pp. 1–30, 2023

    2023

  19. [25]

    Dominance statistics: Ordinal analyses to answer ordi- nal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordi- nal questions.” Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  20. [26]

    Atconf: auto-tuning high dimensional configuration parameters for big data process- ing frameworks,

    H. Dou, K. Wang, Y. Zhang, and P . Chen, “Atconf: auto-tuning high dimensional configuration parameters for big data process- ing frameworks,” Cluster Computing, vol. 26, no. 5, pp. 2737–2755, 2023. 31

    2023

  21. [27]

    Framework and benchmarks for combinatorial and mixed-variable bayesian optimization,

    K. Dreczkowski, A. Grosnit, and H. Bou-Ammar, “Framework and benchmarks for combinatorial and mixed-variable bayesian optimization,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orle...

    2023

  22. [28]

    BOHB: robust and efficient hyperparameter optimization at scale,

    S. Falkner, A. Klein, and F. Hutter, “BOHB: robust and efficient hyperparameter optimization at scale,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , ser. Proceedings of Machine Learn...

    2018

  23. [29]

    To tune or not to tune? in search of optimal configurations for data analytics,

    A. Fekry, L. Carata, T. Pasquier, A. Rice, and A. Hopper, “To tune or not to tune? in search of optimal configurations for data analytics,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 2494– 2504

    2020

  24. [30]

    Resource-guided configuration space reduction for deep learning models,

    Y. Gao, Y. Zhu, H. Zhang, H. Lin, and M. Yang, “Resource-guided configuration space reduction for deep learning models,” in 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021 . IEEE, 2021, pp. 175–

    2021

  25. [31]

    Garnett, Bayesian optimization

    R. Garnett, Bayesian optimization. Cambridge University Press, 2023

    2023

  26. [32]

    Synthesis of probabilistic models for quality-of-service software engineering,

    S. Gerasimou, R. Calinescu, and G. Tamburrelli, “Synthesis of probabilistic models for quality-of-service software engineering,” Autom. Softw. Eng. , vol. 25, no. 4, pp. 785–831, 2018. [Online]. Available: https://doi.org/10.1007/s10515-018-0235-8

    2018 doi

  27. [33]

    Does configuration encoding matter in learning software performance? an empirical study on encoding schemes,

    J. Gong and T. Chen, “Does configuration encoding matter in learning software performance? an empirical study on encoding schemes,” in 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, P A, USA, May 23-24, 2022 . ACM, 2022, pp. 482–4...

    2022

  28. [34]

    Predicting software performance with divide-and-learn,

    ——, “Predicting software performance with divide-and-learn,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , S. Chandra, K. Blinc...

    2023

  29. [35]

    Deep configuration performance learning: A systematic survey and taxonomy,

    ——, “Deep configuration performance learning: A systematic survey and taxonomy,” ACM Transactions on Software Engineering and Methodology, 2024

    2024

  30. [36]

    Predicting configuration performance in multiple environments with sequential meta-learning,

    ——, “Predicting configuration performance in multiple environments with sequential meta-learning,” Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 359–382, 2024. [Online]. Available: https://doi.org/10.1145/3643743

    2024 doi

  31. [37]

    Dividable configuration performance learning,

    J. Gong, T. Chen, and R. Bahsoon, “Dividable configuration performance learning,” IEEE Transactions on Software Engineering, 2024

    2024

  32. [38]

    Are we forgetting about compositional optimisers in bayesian optimisation?

    A. Grosnit, A. I. Cowen-Rivers, R. Tutunov, R. Griffiths, J. Wang, and H. Bou-Ammar, “Are we forgetting about compositional optimisers in bayesian optimisation?” J. Mach. Learn. Res. , vol. 22, pp. 160:1–160:78, 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1422.html

    2021

  33. [39]

    Data-efficient perfor- mance learning for configurable systems,

    J. Guo, D. Yang, N. Siegmund, S. Apel, A. Sarkar, P . Valov, K. Czarnecki, A. Wasowski, and H. Yu, “Data-efficient perfor- mance learning for configurable systems,” Empirical Software Engineering, vol. 23, pp. 1826–1867, 2018

    2018

  34. [40]

    Deepperf: Performance prediction for configurable software with deep sparse neural network,

    H. Ha and H. Zhang, “Deepperf: Performance prediction for configurable software with deep sparse neural network,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1095–1106

    2019

  35. [41]

    Deepperf: performance prediction for configurable software with deep sparse neural network,

    ——, “Deepperf: performance prediction for configurable software with deep sparse neural network,” in Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, J. M. Atlee, T. Bultan, and J. Whittle, Eds. IEEE / ...

    2019

  36. [42]

    An empirical study on performance bugs for highly configurable software systems,

    X. Han and T. Yu, “An empirical study on performance bugs for highly configurable software systems,” in Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8-9, 2016 . ACM, 2016, p...

    2016

  37. [43]

    Sequential model- based optimization for general algorithm configuration,

    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model- based optimization for general algorithm configuration,” in Learning and Intelligent Optimization: 5th International Confer- ence, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5 . Springer, 2011, pp. 507–523

    2011

  38. [44]

    Paramils: an automatic algorithm configuration framework,

    F. Hutter, H. H. Hoos, K. Leyton-Brown, and T. Stützle, “Paramils: an automatic algorithm configuration framework,” Journal of artificial intelligence research, vol. 36, pp. 267–306, 2009

    2009

  39. [45]

    An uncertainty-aware approach to optimal configuration of stream processing systems,

    P . Jamshidi and G. Casale, “An uncertainty-aware approach to optimal configuration of stream processing systems,” in 24th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2016, London, United Kingdom, Septemb...

    2016 doi

  40. [46]

    Transfer learning for performance modeling of configurable systems: an exploratory analysis,

    P . Jamshidi, N. Siegmund, M. Velez, C. Kästner, A. Patel, and Y. Agarwal, “Transfer learning for performance modeling of configurable systems: an exploratory analysis,” in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, U...

    2017

  41. [47]

    Learning to sample: exploiting similarities across environments to learn performance models for configurable systems,

    P . Jamshidi, M. Velez, C. Kästner, and N. Siegmund, “Learning to sample: exploiting similarities across environments to learn performance models for configurable systems,” in Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium o...

    2018

  42. [48]

    Learning to sample: Exploiting similarities across environments to learn performance models for configurable systems,

    P . Jamshidi, M. Velez, C. Kästner, and N. Siegmund, “Learning to sample: Exploiting similarities across environments to learn performance models for configurable systems,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Sympos...

    2018

  43. [49]

    Fitness distance correlation as a measure of problem difficulty for genetic algorithms,

    T. Jones and S. Forrest, “Fitness distance correlation as a measure of problem difficulty for genetic algorithms,” in Proceedings of the 6th International Conference on Genetic Algorithms, July 15-19, 1995 , 1995, pp. 184–192

    1995

  44. [50]

    Llamatune: sample-efficient dbms configura- tion tuning,

    K. Kanellis, C. Ding, B. Kroth, A. Müller, C. Curino, and S. Venkataraman, “Llamatune: sample-efficient dbms configura- tion tuning,” arXiv preprint arXiv:2203.05128, 2022

    2022 arXiv

  45. [51]

    Search dynamics on multimodal multiobjective problems,

    P . Kerschke, H. Wang, M. Preuss, C. Grimme, A. H. Deutz, H. Trautmann, and M. T. M. Emmerich, “Search dynamics on multimodal multiobjective problems,” Evol. Comput., vol. 27, no. 4, pp. 577–609, 2019. [Online]. Available: https://doi.org/10.1162/evco_a_00234

    2019 doi

  46. [52]

    Robotune: high-dimensional configura- tion tuning for cluster-based data analytics,

    M. M. Khan and W. Yu, “Robotune: high-dimensional configura- tion tuning for cluster-based data analytics,” in Proceedings of the 50th international conference on parallel processing, 2021, pp. 1–10

    2021

  47. [53]

    Scope: Safe exploration for dynamic computer systems opti- mization,

    H. Kim, A. Pervaiz, H. Hoffmann, M. Carbin, and Y. Ding, “Scope: Safe exploration for dynamic computer systems opti- mization,” arXiv preprint arXiv:2204.10451, 2022

    2022 arXiv

  48. [54]

    Whence to learn? transferring knowledge in configurable systems using beetle,

    R. Krishna, V . Nair, P . Jamshidi, and T. Menzies, “Whence to learn? transferring knowledge in configurable systems using beetle,” IEEE Transactions on Software Engineering, 2020

    2020

  49. [55]

    Whence to learn? transferring knowledge in configurable systems using BEETLE,

    ——, “Whence to learn? transferring knowledge in configurable systems using BEETLE,” IEEE Trans. Software Eng. , vol. 47, no. 12, pp. 2956–2972, 2021. [Online]. Available: https: //doi.org/10.1109/TSE.2020.2983927

    2021

  50. [56]

    Conex: Efficient exploration of big-data system configurations for better perfor- mance,

    R. Krishna, C. Tang, K. Sullivan, and B. Ray, “Conex: Efficient exploration of big-data system configurations for better perfor- mance,” IEEE Transactions on Software Engineering , vol. 48, no. 3, pp. 893–909, 2020

    2020

  51. [57]

    K2vtune: A workload- aware configuration tuning for rocksdb,

    J. Lee, S. Seo, J. Choi, and S. Park, “K2vtune: A workload- aware configuration tuning for rocksdb,” Inf. Process. Manag. , vol. 61, no. 1, p. 103567, 2024. [Online]. Available: https: //doi.org/10.1016/j.ipm.2023.103567 32

    2024

  52. [58]

    Understanding the automated parameter optimization on transfer learning for cross-project defect prediction: an empirical study,

    K. Li, Z. Xiang, T. Chen, S. Wang, and K. C. Tan, “Understanding the automated parameter optimization on transfer learning for cross-project defect prediction: an empirical study,” in ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June ...

    2020

  53. [59]

    Hyperband: A novel bandit-based approach to hyperparameter optimization,

    L. Li, K. G. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” J. Mach. Learn. Res. , vol. 18, pp. 185:1–185:52, 2017. [Online]. Available: http://jmlr.org/papers/ v18/16-558.html

    2017

  54. [60]

    Phronesis: Efficient performance modeling for high-dimensional configuration tuning,

    Y. Li and B. C. Lee, “Phronesis: Efficient performance modeling for high-dimensional configuration tuning,” ACM Trans. Archit. Code Optim. , vol. 19, no. 4, pp. 56:1–56:26, 2022. [Online]. Available: https://doi.org/10.1145/3546868

    2022 doi

  55. [61]

    An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks,

    L. Liao, H. Li, W. Shang, and L. Ma, “An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 3, pp. 53:1–53:40, 2022. [Online]. Available: https://...

    2022 doi

  56. [62]

    Classification and regression by randomforest,

    A. Liaw, M. Wiener et al. , “Classification and regression by randomforest,” R news, vol. 2, no. 3, pp. 18–22, 2002

    2002

  57. [63]

    The irace package: Iterated racing for automatic algorithm configuration,

    M. López-Ibáñez, J. Dubois-Lacoste, L. P . Cáceres, M. Birattari, and T. Stützle, “The irace package: Iterated racing for automatic algorithm configuration,” Operations Research Perspectives , vol. 3, pp. 43–58, 2016

    2016

  58. [64]

    A comparison of performance specialization learning for configurable systems,

    H. Martin, M. Acher, J. A. Pereira, and J. Jézéquel, “A comparison of performance specialization learning for configurable systems,” in SPLC ’21: 25th ACM International Systems and Software Product Line Conference, Leicester, United Kingdom, September 6-11, 2021, Volume A , M....

    2021

  59. [65]

    Kruskal-wallis test,

    P . E. McKight and J. Najab, “Kruskal-wallis test,” The corsini encyclopedia of psychology, pp. 1–1, 2010

    2010

  60. [66]

    Montgomery, E

    D. Montgomery, E. Peck, and G. Vining, Introduction to Linear Regression Analysis, ser. Wiley Series in Probability and Statistics. Wiley, 2021. [Online]. Available: https://books.google.com/ books?id=tCIgEAAAQBAJ

    2021

  61. [67]

    Analyzing the im- pact of workloads on modeling the performance of configurable software systems,

    C. S. Mühlbauer, F. Sattler, and N. Siegmund, “Analyzing the im- pact of workloads on modeling the performance of configurable software systems,” in Proceedings of the International Conference on Software Engineering (ICSE), IEEE, 2023

    2023

  62. [68]

    Spearman correlation coefficients, differences between,

    L. Myers and M. J. Sirois, “Spearman correlation coefficients, differences between,” Encyclopedia of statistical sciences , vol. 12, 2004

    2004

  63. [69]

    Using bad learners to find good configurations,

    V . Nair, T. Menzies, N. Siegmund, and S. Apel, “Using bad learners to find good configurations,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017 , E. Bodden, W. Schäfer, A. van Deurse...

    2017

  64. [70]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,” IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2018

    2018

  65. [71]

    Finding faster configurations using FLASH,

    ——, “Finding faster configurations using FLASH,” IEEE Trans. Software Eng. , vol. 46, no. 7, pp. 794–811, 2020. [Online]. Available: https://doi.org/10.1109/TSE.2018.2870895

    2020

  66. [72]

    Veer: Disagreement-free multi-objective configuration,

    K. Peng, C. Kaltenecker, N. Siegmund, S. Apel, and T. Menzies, “Veer: Disagreement-free multi-objective configuration,” CoRR, 2021

    2021

  67. [73]

    VEER: disagreement-free multi-objective configuration,

    ——, “VEER: disagreement-free multi-objective configuration,” CoRR, vol. abs/2106.02716, 2021. [Online]. Available: https: //arxiv.org/abs/2106.02716

    2021 arXiv

  68. [74]

    Learning software configuration spaces: A systematic literature review,

    J. A. Pereira, M. Acher, H. Martin, J. Jézéquel, G. Botterweck, and A. Ventresque, “Learning software configuration spaces: A systematic literature review,” J. Syst. Softw. , vol. 182, p. 111044, 2021. [Online]. Available: https://doi.org/10.1016/j.jss. 2021.111044

    2021

  69. [75]

    A comprehensive survey on fitness landscape analysis,

    E. Pitzer and M. Affenzeller, “A comprehensive survey on fitness landscape analysis,” in Recent Advances in Intelligent Engineering Systems , ser. Studies in Computational Intelligence, J. C. Fodor, R. Klempous, and C. P . S. Araujo, Eds. Springer, 2012, vol. 378, pp. 161–191....

    2012 doi

  70. [76]

    C. E. Rasmussen, C. K. Williams et al. , Gaussian processes for machine learning. Springer, 2006, vol. 1

    2006

  71. [77]

    Software configuration engineering in practice interviews, survey, and systematic literature review,

    M. Sayagh, N. Kerzazi, B. Adams, and F. Petrillo, “Software configuration engineering in practice interviews, survey, and systematic literature review,” IEEE Trans. Software Eng. , vol. 46, no. 6, pp. 646–673, 2020. [Online]. Available: https://doi.org/10.1109/TSE.2018.2867847

    2020

  72. [78]

    On parameter tuning in search based software engineering: A replicated empirical study,

    A. S. Sayyad, K. Goseva-Popstojanova, T. Menzies, and H. Am- mar, “On parameter tuning in search based software engineering: A replicated empirical study,” in 2013 3rd International Workshop on Replication in Empirical Software Engineering Research . IEEE, 2013, pp. 84–90

    2013

  73. [79]

    Greening large language models of code,

    J. Shi, Z. Yang, H. J. Kang, B. Xu, J. He, and D. Lo, “Greening large language models of code,” International Conference on Software Engineering (ICSE), 2024

    2024

  74. [80]

    Spl conqueror: Toward optimization of non-functional properties in software product lines,

    N. Siegmund, M. Rosenmüller, M. Kuhlemann, C. Kästner, S. Apel, and G. Saake, “Spl conqueror: Toward optimization of non-functional properties in software product lines,” Software Quality Journal, vol. 20, pp. 487–517, 2012

    2012

  75. [81]

    NAPEL: near- memory computing application performance prediction via ensemble learning,

    G. Singh, J. Gómez-Luna, G. Mariani, G. F. Oliveira, S. Corda, S. Stuijk, O. Mutlu, and H. Corporaal, “NAPEL: near- memory computing application performance prediction via ensemble learning,” in Proceedings of the 56th Annual Design Automation Conference 2019, DAC 2019, Las Ve...

    2019

  76. [82]

    A tutorial on support vector regression,

    A. J. Smola and B. Schölkopf, “A tutorial on support vector regression,” Statistics and computing, vol. 14, pp. 199–222, 2004

    2004

  77. [83]

    Decision tree methods: applications for classification and prediction,

    Y.-Y. Song and Y. Lu, “Decision tree methods: applications for classification and prediction,” Shanghai archives of psychiatry , vol. 27, no. 2, p. 130—135, April 2015. [Online]. Available: https://europepmc.org/articles/PMC4466856

    2015

  78. [84]

    Landscapes and their correlation functions,

    P . F. Stadler, “Landscapes and their correlation functions,” Journal of Mathematical chemistry, vol. 20, no. 1, pp. 1–45, 1996

    1996

  79. [85]

    An empirical comparison of model validation tech- niques for defect prediction models,

    C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Mat- sumoto, “An empirical comparison of model validation tech- niques for defect prediction models,” no. 1, 2017

    2017

  80. [86]

    The impact of automated parameter optimization on defect prediction models,

    ——, “The impact of automated parameter optimization on defect prediction models,” IEEE Trans. Software Eng. , vol. 45, no. 7, pp. 683–711, 2019. [Online]. Available: https://doi.org/10.1109/TSE.2018.2794977

    2019

  81. [87]

    Multidimensional knapsack problem: A fitness landscape analysis,

    J. Tavares, F. B. Pereira, and E. Costa, “Multidimensional knapsack problem: A fitness landscape analysis,” IEEE Trans. Syst. Man Cybern. Part B , vol. 38, no. 3, pp. 604–616,

  82. [88]

    Auto- matic database management system tuning through large-scale machine learning,

    D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang, “Auto- matic database management system tuning through large-scale machine learning,” in Proceedings of the 2017 ACM international conference on management of data, 2017, pp. 1009–1024

    2017

  83. [89]

    An inquiry into machine learning-based auto- matic configuration tuning services on real-world database man- agement systems,

    D. Van Aken, D. Yang, S. Brillard, A. Fiorino, B. Zhang, C. Bilien, and A. Pavlo, “An inquiry into machine learning-based auto- matic configuration tuning services on real-world database man- agement systems,” Proceedings of the VLDB Endowment , vol. 14, no. 7, pp. 1241–1253, 2021

    2021

  84. [90]

    Configcrusher: towards white-box performance analysis for configurable systems,

    M. Velez, P . Jamshidi, F. Sattler, N. Siegmund, S. Apel, and C. Kästner, “Configcrusher: towards white-box performance analysis for configurable systems,” Autom. Softw. Eng. , vol. 27, no. 3, pp. 265–300, 2020. [Online]. Available: https://doi.org/10. 1007/s10515-020-00273-8

    2020

  85. [93]

    A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering,

    S. Wang, S. Ali, T. Yue, Y. Li, and M. Liaaen, “A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering,” in Proceedings of the 38th International Conference on Software Engineering, ICSE 33 2016, Austin,...

    2016

  86. [94]

    Giving back: Contributions congruent to library dependency changes in a software ecosystem,

    S. Wattanakriengkrai, D. Wang, R. G. Kula, C. Treude, P . Thongtanunam, T. Ishio, and K. Matsumoto, “Giving back: Contributions congruent to library dependency changes in a software ecosystem,” IEEE Trans. Software Eng., vol. 49, no. 4, pp. 2566–2579, 2023. [Online]. Available...

    2023

  87. [95]

    Twins or false friends? a study on energy consumption and performance of configurable software,

    M. Weber, C. Kaltenecker, F. Sattler, S. Apel, and N. Siegmund, “Twins or false friends? a study on energy consumption and performance of configurable software,” in Proceedings of the 45th International Conference on Software Engineering, ICSE , vol. 23, 2023, p. 12

    2023

  88. [96]

    Twins or false friends? A study on energy consumption and performance of configurable software,

    ——, “Twins or false friends? A study on energy consumption and performance of configurable software,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023, pp. 2098–

    2023

  89. [97]

    Hyperparameter optimization for effort estimation,

    T. Xia, R. Krishna, J. Chen, G. Mathew, X. Shen, and T. Menzies, “Hyperparameter optimization for effort estimation,” CoRR, vol. abs/1805.00336, 2018. [Online]. Available: http: //arxiv.org/abs/1805.00336

    2018 arXiv

  90. [98]

    Locat: Low-overhead online con- figuration auto-tuning of spark sql applications,

    J. Xin, K. Hwang, and Z. Yu, “Locat: Low-overhead online con- figuration auto-tuning of spark sql applications,” in Proceedings of the 2022 International Conference on Management of Data , 2022, pp. 674–684

    2022

  91. [99]

    Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software,

    T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, and R. Talwadker, “Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/F...

    2015

  92. [100]

    Distilled lifelong self-adaptation for configurable systems,

    Y. Ye, T. Chen, and M. Li, “Distilled lifelong self-adaptation for configurable systems,” International Conference on Software Engineering (ICSE), 2025

    2025

  93. [101]

    Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing,

    Z. Yu, Z. Bei, and X. Qian, “Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing,” in Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2018, Williamsb...

    2018

  94. [102]

    Facilitating database tuning with hyper-parameter optimization: A comprehensive experimental evaluation,

    X. Zhang, Z. Chang, Y. Li, H. Wu, J. Tan, F. Li, and B. Cui, “Facilitating database tuning with hyper-parameter optimization: A comprehensive experimental evaluation,” Proc. VLDB Endow. , vol. 15, no. 9, pp. 1808–1821, 2022. [Online]. Available: https://www.vldb.org/pvldb/vol1...

    2022

  95. [103]

    Facilitating database tuning with hyper-parameter opti- mization: A comprehensive experimental evaluation,

    ——, “Facilitating database tuning with hyper-parameter opti- mization: A comprehensive experimental evaluation,” 2022

    2022

  96. [104]

    Restune: Resource oriented tuning boosted by meta- learning for cloud databases,

    X. Zhang, H. Wu, Z. Chang, S. Jin, J. Tan, F. Li, T. Zhang, and B. Cui, “Restune: Resource oriented tuning boosted by meta- learning for cloud databases,” in Proceedings of the 2021 interna- tional conference on management of data , 2021, pp. 2102–2114

    2021

  97. [105]

    Towards dynamic and safe configuration tuning for cloud databases,

    X. Zhang, H. Wu, Y. Li, J. Tan, F. Li, and B. Cui, “Towards dynamic and safe configuration tuning for cloud databases,” inProceedings of the 2022 International Conference on Management of Data , 2022, pp. 631–645

    2022

  98. [106]

    An evolutionary study of configuration design and implementation in cloud systems,

    Y. Zhang, H. He, O. Legunsen, S. Li, W. Dong, and T. Xu, “An evolutionary study of configuration design and implementation in cloud systems,” in 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021 . IEEE, 2021, pp. 188–200. ...

    2021

  99. [107]

    Automatic database knob tuning: A survey,

    X. Zhao, X. Zhou, and G. Li, “Automatic database knob tuning: A survey,” IEEE Transactions on Knowledge and Data Engineering , 2023

    2023

  100. [109]

    Active learning for multi-objective optimization,

    M. Zuluaga, G. Sergent, A. Krause, and M. Püschel, “Active learning for multi-objective optimization,” in Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , ser. JMLR Workshop and Conference Proceedings, vol. 28...

    2013

  101. [187]

    Available: https://doi.org/10.1109/ICSE43902

    [Online]. Available: https://doi.org/10.1109/ICSE43902. 2021.00028

    2021

  102. [2008]

    Available: https://doi.org/10.1109/TSMCB.2008

    [Online]. Available: https://doi.org/10.1109/TSMCB.2008. 915539

    2008 doi

  103. [2021]

    Available: http://www.vldb.org/pvldb/vol14/ p1401-cereda.pdf

    [Online]. Available: http://www.vldb.org/pvldb/vol14/ p1401-cereda.pdf

  104. [2024]

    Available: https://doi.org/10.1145/3643751

    [Online]. Available: https://doi.org/10.1145/3643751

    doi

  105. [2110]

    Available: https://doi.org/10.1109/ICSE48619

    [Online]. Available: https://doi.org/10.1109/ICSE48619. 2023.00177

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.