{"id":"35cecbb6-6f6d-4970-809b-04aa14fba6d0","arxiv_id":"2501.01876","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Across 29 systems, 10 models and 17 tuners, higher surrogate-model accuracy frequently fails to improve, and sometimes degrades, configuration tuning quality.","lead":"A large study of software configuration tuning finds that more accurate prediction models often do not improve the tuning outcome, and can even make it worse. The finding challenges the field's usual assumption that model accuracy is the key to good tuning, and points to better ways of judging surrogate models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper operationalizes 'accuracy' as aggregate test-set error, but the 'accuracy is all' belief in the community often targets accuracy near the optimum or on top-ranked candidates; if that is the real belief, the central refutation may miss its target.","rationale":"The reader's weakest_assumption identifies exactly this mismatch between the operationalized accuracy (aggregate MAPE/muRD on random test sets) and the community's notion of accuracy when asserting 'accuracy is all.' This is the most load-bearing concern because the entire paper's headline conclusion -- that the community should step back from the 'accuracy is all' belief -- depends on the choice of accuracy metric. If the belief is really about useful/local accuracy, the paper's data may be consistent with that belief, and the contribution becomes an argument about metric choice rather than a refutation of the belief. The concern is concrete and testable: the authors have the full dataset and can recompute the analyses with locally focused or exploration-weighted accuracy metrics. The paper is otherwise well-executed: the scale is large, the statistical methods are sound (Mann-Whitney U, Scott-Knott ESD, Spearman correlation with significance), and the landscape analysis provides a plausible qualitative explanation. The reader's CONDITIONAL verdict already reflects this concern, so my assessment does not move the verdict; it reinforces the need for the conditional test before accepting the strong interpretation.","tokens_in":66865,"tokens_out":3566,"duration_ms":40764,"concrete_test":"Re-run the RQ3-RQ5 analyses on a representative subset (e.g., 10 systems spanning small/large spaces) using three alternative accuracy definitions: (1) MAPE computed only on the top 10% of configurations by measured performance (near-optimum accuracy); (2) ranking accuracy restricted to the top-10 candidates (e.g., overlap or Kendall tau between true and predicted top-10); (3) accuracy evaluated on the configurations the tuner actually measured during tuning (exploration-weighted error). If any of these metrics yields a median Spearman correlation with tuning quality above 0.4 or best/worst match rates above 50% across the subset, the paper's conclusion should be narrowed to 'aggregate accuracy is all' rather than 'accuracy is all.' If all remain as weak as the aggregate metrics, the central claim is robust to this concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper defines the 'accuracy is all' belief broadly (Section 2.4) but measures model accuracy exclusively as MAPE and 1/n x muRD on a randomly held-out test set (Sections 3.5.1-3.5.2). The belief, as supported by the paper's own quotes, is not necessarily about aggregate prediction error: Zhu et al. [108] complain that 'Gaussian Process's predictions about best points are hardly accurate,' and Wang et al. [92] claim the model can 'accurately fit the target.' These statements concern accuracy in the regions that matter for tuning (near the optimum, or on configurations the tuner will actually evaluate), not average error over a random sample of the configuration space. If practitioners care about local accuracy or the ranking of top candidates, then the finding that aggregate MAPE/muRD correlates weakly with tuning quality (RQ3-RQ4) does not refute the belief that better models -- in the sense of better predictions where it counts -- improve tuning. The paper's own landscape analysis (Section 5.3.2) shows that large aggregate-accuracy improvements can leave the relevant landscape structure unchanged, but it does not test whether a locally oriented accuracy metric would align better with tuning quality. Thus the central message ('take one step back from accuracy is all') overgeneralizes from aggregate accuracy to accuracy as a whole; the conclusion is load-bearing on this operationalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a large-scale empirical study of how surrogate-model accuracy affects configuration tuning quality. It covers 10 models, 17 tuners, 29 systems, and four metrics, leading to 13,612 investigation cases, each repeated 30 times. The study answers five research questions: whether models help tuning quality (RQ1), whether originally chosen models are best for tuning (RQ2), whether the most/least accurate model yields the best/worst tuning quality (RQ3), how accuracy and tuning quality correlate (RQ4), and how much accuracy change is needed for meaningful tuning improvement (RQ5). The central claim is that the community's \"accuracy is all\" belief is misleading: higher aggregate model accuracy frequently does not improve tuning quality (up to 58% of cases) and can even degrade it (up to 24% of cases). The paper also proposes fitness-landscape metrics as an alternative lens for assessing model usefulness. The manuscript is careful statistically, using Mann-Whitney U-tests, Scott-Knott ESD, Spearman correlation, and a test-size validation on the five largest systems, and it provides a public repository for code and data.","tokens_in":67153,"tokens_out":5622,"duration_ms":59844,"significance":"If the central claim holds, this is an important result for model-based configuration tuning: it challenges the widespread practice of judging surrogate models mainly by aggregate prediction error, and it provides concrete evidence that the connection between model accuracy and tuning quality is weak and sometimes negative. The study's scale (29 systems, 10 models, 17 tuners) and its 30-run repetition are genuinely strong, and the paper is transparent about statistical procedures and open-science artifacts. The landscape-analysis viewpoint is a valuable new interpretation even if it is only presented on selected examples. However, the significance is tempered by a gap between the belief as stated in the motivating quotations and the accuracy metrics actually measured: the paper operationalizes accuracy as MAPE and 1/n x muRD on randomly held-out test sets, while the community quotations the paper itself cites concern accuracy near optima or on \"best points.\" The empirical results therefore establish a claim about aggregate test-set accuracy rather than about the local or decision-oriented accuracy that may be what practitioners actually mean when they prefer \"more accurate\" models.","major_comments":[{"comment":"The paper defines the \"accuracy is all\" belief in broad terms but measures accuracy exclusively as MAPE and 1/n x muRD on randomly held-out test sets. The motivating quotations, however, refer to accuracy in the regions that matter for tuning: Zhu et al. [108] complain that \"Gaussian Process's predictions about best points are hardly accurate,\" and Wang et al. [92] claim the model can \"accurately fit the target.\" These are claims about local or top-ranked accuracy, not about average error over a random sample of the configuration space. If practitioners care about accuracy near the optimum or about the ranking of top candidates, the paper's RQ3-RQ5 results show only that aggregate accuracy correlates weakly with tuning quality; they do not refute a belief about locally relevant accuracy. Please either add accuracy metrics evaluated on tuning-relevant configurations (e.g., error on top-k predicted configurations or near-optimum configurations) and re-run the analyses, or explicitly narrow the central claim to \"aggregate accuracy can lie.\"","section":"Sections 2.4 and 3.5.1-3.5.2"},{"comment":"The comparison in RQ3 and RQ4 is across ten different surrogate models, so model accuracy is confounded with model family, uncertainty-estimation mechanism, and interaction with the acquisition function. The causal wording in the abstract (\"higher accuracy actually leads to no improvement\" and \"accuracy can lie\") is not supported by Spearman correlations computed across different models, because a difference in accuracy between, say, a decision tree and a deep network is inseparable from many other differences between the models. I recommend either rephrasing the findings as associational/predictive claims about the usefulness of aggregate accuracy as a proxy, or adding a controlled analysis in which accuracy is varied within a fixed model family (e.g., through regularization strength, training-set size, or label noise) while the model class and the tuner remain unchanged.","section":"Sections 4.3-4.4 and abstract"},{"comment":"The \"minimum accuracy change\" in RQ5 is estimated by selecting, for each anchor model, the smallest accuracy difference to any significantly better model, using many pairwise Mann-Whitney tests without multiple-comparison correction. The anchors are then binned by their own accuracy, and the reported mean and standard deviation of Delta are aggregated over all tuner-system combinations. The interpretation of Finding 6 depends on the number of samples per bin and on how often no significantly better model exists in a bin; please report per-bin sample sizes and the false-positive risk from the multiple comparisons, or the reported ranges such as [0.6, 85.9] may be taken to be more precise than the procedure warrants.","section":"Section 4.5.1 and Figures 9-10"},{"comment":"The landscape explanation is based on a small number of hand-picked examples (TOMCAT, LLVM, HSMGP, HSQLDB, SQLITE). The text explicitly says these are representative examples, but the broader conclusion that landscape properties explain why accuracy changes do not translate into tuning-quality changes would be substantially strengthened if the landscape metrics (FDC, correlation length, global-optimum deviation) were computed systematically across the full set of system-tuner-model combinations and compared against the RQ3-RQ5 outcomes. As presented, the landscape analysis is an exploratory interpretation rather than a tested explanation. Please add such a systematic analysis or soften the contribution claim to \"observational case studies.\"","section":"Section 5 (especially 5.3.2 and 5.3.3)"}],"minor_comments":[{"comment":"The text refers to \"MPAE\" in the paragraph on MAPE; this appears to be a typo for MAPE.","section":"Section 3.5.1"},{"comment":"The sentence ends \"prevnting the samples from being concentrated in certain parts\"; please correct the typo \"prevnting.\"","section":"Section 3.4.1"},{"comment":"The efficiency metric r = m/b is undefined when a tuner-model pair never reaches the baseline performance T within the budget; please specify how such cases are handled, because they occur in several reported results.","section":"Section 3.5.3"},{"comment":"The caption states \"0.0 +/- 0.0 implies no samples can significantly improve the tuning for a range,\" but a zero mean with zero standard deviation could also occur if exactly one sample has Delta = 0; please clarify whether the entry means \"no samples\" or \"a zero Delta.\"","section":"Section 4.5.2 / Figure 10 caption"},{"comment":"The statement that rho = 0 means the two metrics \"do not correlate with each other in any way\" is too strong for Spearman's rank correlation, which only detects monotonic association; please rephrase.","section":"Section 3.6.3"},{"comment":"The \"Scale\" column uses \"Small\" and \"Large\" to classify systems by search space; the paper's introduction of these terms in Section 3.2.1 should be referenced at the table caption for clarity.","section":"Table 1"},{"comment":"The computation of the 13,612 total is described in a long parenthetical equation; a small table or bullet list showing how the 5,220 accuracy cases and 8,392 tuning-quality cases are constructed would make the case count easier to verify.","section":"Section 3.7"}],"recommendation":"major_revision","confidential_remarks":"This is a substantial empirical study with strong infrastructure and careful statistical practice. My main concern is that the central message is broader than what the accuracy metrics support: the paper measures aggregate MAPE and rank difference on random test sets, while the motivating quotations concern accuracy near optima. I would be willing to see a revision that either adds a tuning-relevant accuracy metric or clearly restricts the conclusion to aggregate accuracy. The causal wording in the abstract and Section 4 should also be softened unless a controlled accuracy-manipulation experiment is added. The paper is within the journal's scope and, with these changes, could make an important contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Couple of things you should know about this one. It's the largest empirical study I've seen on whether more accurate surrogate models actually improve configuration tuning: 10 models, 17 tuners, 29 systems, 13,612 cases, 30 runs each. The headline finding—that a more accurate model (lower MAPE or muRD) often fails to improve tuning quality, and sometimes makes it worse—is real and well-supported. The statistical hygiene is above the usual bar: Mann-Whitney U with significance thresholds, Scott-Knott ESD ranking, Spearman correlations, and a check that their test-sample size doesn't bias accuracy. The RQ5 thresholds (e.g., a MAPE in [30,40] needs ~13% improvement to matter) are genuinely useful, even if the variance is large. The landscape analysis gives a plausible reason why: aggregate accuracy can change a lot while leaving the fitness landscape structure that tuners actually navigate mostly unchanged.\n\nThe soft spot is the one the stress-test note flags. Accuracy is operationalized as aggregate prediction error over a random held-out test set. But the 'accuracy is all' belief, as the paper's own quotes show, is often about accuracy where it counts—near the optimum, on the top candidates the tuner evaluates. Zhu et al. complain that GP's predictions about the best points are hardly accurate; Wang et al. say their model can accurately fit the target. Those are local claims. The paper never tests whether a locally-oriented accuracy metric (e.g., error on top-k predicted configurations or near the optimum) correlates better with tuning quality. So the conclusion 'take one step back from accuracy is all' is solid for aggregate accuracy but overgeneralized for accuracy as a whole. This doesn't sink the paper; it narrows it. A revision that adds a local accuracy analysis would substantially strengthen it.\n\nOther soft spots are minor. The RQ5 numbers have wide confidence intervals (the 13.3% figure comes with a ±7.8 standard deviation) and shouldn't be read as precise engineering thresholds. The efficiency metric is a bit convoluted. The artifact repo exists but I couldn't execute it independently here; that's a normal limitation for a study this size.\n\nBottom line: this is a serious, carefully executed empirical paper with a message that the configuration tuning community needs to hear. The central claim about aggregate accuracy holds up. It deserves a real peer review, and I'd want the reviewers to push on the accuracy operationalization before publication.\n\nRecommendation: send it out. I'd probably cite it, and I'd bring it to the reading group.","headline":"A large, statistically careful empirical study that convincingly shows aggregate model accuracy is a poor predictor of tuning quality, though it stops short of refuting accuracy-as-a-whole because it never tests local accuracy.","tokens_in":67662,"tokens_out":2513,"would_cite":true,"duration_ms":24168,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that higher surrogate-model accuracy often fails to improve—and can actively degrade—configuration tuning quality, a claim tested on 13,612 cases across 10 models, 17 tuners, and 29 systems.","keywords":["configuration tuning","surrogate model","model accuracy","software configuration","model-based optimization","fitness landscape analysis","performance prediction","configurable systems"],"falsifier":"Recompute model accuracy on the subset of test configurations closest to the true optimum (for example within a small Hamming distance) or on the top-ranked predicted configurations: if this near-optimum accuracy correlates strongly and positively (Spearman $\\rho > 0.7$) with the achieved tuning quality for a majority of the 29 systems, then the paper's conclusion would be shown to be an artifact of whole-space aggregate accuracy rather than a failure of accuracy as such.","tokens_in":66655,"feed_emoji":"⚙️","tokens_out":15835,"duration_ms":136085,"temperature":0.7,"pith_summary":"This paper sets out to test a widely held belief in software configuration tuning: that a more accurate surrogate model necessarily produces better tuning outcomes. Using an empirical corpus of 13,612 cases covering 10 models, 17 tuners, and 29 systems, it finds that higher accuracy yields no statistically significant improvement in tuning quality in up to 58% of cases under some settings, and actually degrades quality in up to 24%. The paper's central message is that aggregate prediction error, measured by MAPE or a rank-difference metric, is not a reliable proxy for a model's usefulness to a tuner, and that the community should stop treating accuracy as the dominant yardstick for model-based configuration tuning. A reader should care because model accuracy currently drives how configuration-performance models are published, compared, and criticized; if the claim holds, the evaluation criteria and research effort in this area need rethinking.","feed_headline":"58% of tuning cases see no gain from higher model accuracy","feed_subtitle":"A 13,612-case study of 29 systems challenges the 'accuracy is all' belief in model-based configuration tuning.","key_machinery":"The load-bearing apparatus has three parts. First, the measurement corpus: two accuracy metrics (mean absolute percentage error, $\\text{MAPE}=\\frac{1}{n}\\sum_{i=1}^{n}\\left|\\frac{y_i-\\hat{y}_i}{y_i}\\right|\\cdot 100\\%$, and a sample-size-standardized rank difference $\\frac{1}{n}\\times \\mu RD$), two tuning-quality metrics (the measured performance of the best returned configuration and an efficiency ratio of measurements needed to reach a baseline), and 13,612 cases built from 10 models, 17 tuners, and 29 systems with 30 repeated runs each. Second, the statistical apparatus—Mann–Whitney U-tests, Scott–Knott effect-size difference ranking, and Spearman's rank correlation $\\rho$—that converts raw results into win/tie/lose counts and correlation strengths. Third, the interpretive machinery borrowed from fitness landscape analysis: fitness distance correlation $\\varrho$ (the correlation between performance and distance to the nearest global optimum), correlation length $\\ell$ (the autocorrelation of neighboring configurations' performances, measuring ruggedness), and the deviation between the model-predicted and true global optimum in both Hamming distance and performance. This third part carries the explanation: a tuner explores the landscape the model emulates, and models with similar error can inhabit very different landscapes (or vice versa), so error alone cannot predict the tuning outcome.","core_discovery":"On the paper's own terms, the discovery is that 'accuracy can lie': the correlation between a surrogate model's aggregate test-set accuracy and the quality of the configuration the tuner finds is weak, negligible, or even negative in the large majority of cases. Across the corpus, the most accurate model is the best for tuning quality in only 14–45% of cases depending on the metric combination, and the least accurate model marks the worst tuning outcomes in a comparably small share; in up to 97% of cases the accuracy–quality correlation is negligible or negative. The study also finds that the model originally packaged inside a sequential model-based tuner is sub-optimal in about 46% of cases, and that batch-style tuners whose pre-trained model is never updated lose to model-free tuning on performance in 28 of 29 systems. It further quantifies the minimum accuracy change needed to move tuning quality significantly, a threshold that depends on the accuracy range (a model at 30–40% MAPE needs roughly 13% improvement, while a model below 10% MAPE needs under 1%). The offered explanation is that tuning quality is governed by the configuration landscape the model emulates—its guidance toward the optimum, its ruggedness, and the location of its predicted optimum—none of which aggregate accuracy captures.","pith_inferences":["If the accuracy–quality gap stems from the optimizer exploiting the emulated landscape rather than the error surface, then the same pattern should appear in other surrogate-guided search settings, such as hyperparameter optimization or simulation-based design; the paper does not make this generalization, but its mechanism implies it.","A testable extension would recompute accuracy on the test configurations the tuner actually visited, or on those nearest to the predicted optimum; if that 'local accuracy' correlates strongly with tuning quality, the right measurement is not error in general but error where the optimizer operates.","The landscape metrics used for post-hoc explanation could plausibly become a pre-deployment model-selection rule: pick the model whose emulated landscape best matches the measured landscape in fitness-distance correlation and correlation length before spending any tuning budget.","The paper's thresholds for minimum meaningful accuracy change could serve as a community benchmark, making a newly reported accuracy gain interpretable only in comparison with these figures."],"forward_implications":["Model-based configuration tuning should stop using aggregate accuracy as the primary criterion for judging surrogate models, and evaluation should add landscape-based proxies such as guidance toward the optimum, ruggedness, and predicted-optimum deviation.","Claims of a percentage accuracy improvement should be benchmarked against the minimum meaningful accuracy change documented here, for example roughly 13% for a model in the 30–40% MAPE range, before being presented as useful for tuning.","Because the originally chosen model is sub-optimal in about 46% of cases and the best model only becomes clear during tuning, automated joint selection of the model–tuner pair is a natural bi-level optimization problem for future tuners.","Batch model-based tuners that pair a fixed pre-trained model with a model-free search should not be assumed to retain tuning quality: under a fair budget they lose to model-free tuners on performance in 28 of 29 systems, whereas sequentially updated models beat the model-free baseline in 16 of 29 systems on performance and up to 21 of 29 on efficiency."],"supporting_citations":[{"why":"Supplies a state-of-the-art highly accurate performance model and its accuracy-race claim; the model is part of the corpus whose tuning impact is measured.","marker":"[24]"},{"why":"Defines the MAPE accuracy convention and contributes DeepPerf, a sparse deep model in the corpus; its accuracy-based framing is a target of the study.","marker":"[40]"},{"why":"Contributes the rank-difference accuracy metric (µRD) and the FLASH sequential tuner, and its quoted accuracy-based explanation of tuning success is under test.","marker":"[70]"},{"why":"Provides DECART, a data-efficient performance model in the corpus, representing the accuracy-driven modeling line.","marker":"[39]"},{"why":"Its quoted claim that accurate model fitting explains rapid tuning success exemplifies the 'accuracy is all' belief the study challenges.","marker":"[92]"},{"why":"Its quoted claim that accuracy improvement implies greater usefulness for tuning typifies the belief the empirical results contradict.","marker":"[101]"},{"why":"Its quoted criticism of model inaccuracy motivates the belief and supplies the BestConfig model-free baseline.","marker":"[108]"},{"why":"Supplies a sequential model-based tuner (BOCA) and the hot-start practice used in the experimental protocol.","marker":"[11]"},{"why":"Supplies the fitness landscape analysis concepts (guidance, ruggedness) used to explain why accuracy fails to predict tuning quality.","marker":"[75]"}],"fun_headline_variants":["Accuracy can lie: higher surrogate precision rarely tunes better","Surrogate accuracy doesn't guarantee better config tuning","Large-scale study: model accuracy often misleads tuners","Why accurate surrogate models still fail at tuning","97% of the time, surrogate accuracy fails to aid tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study defines 'model accuracy' as aggregate prediction error (MAPE or a rank-difference measure) computed on a random held-out test set; if the accuracy the community actually relies on when preferring a model is something different, such as accuracy near the optimum or the ability to rank the top candidates, then the conclusion that accuracy can lie holds for the aggregate measure and not necessarily for the accuracy that matters.","fun_headline_variants_meta":{"raw":{"variants":["Accuracy can lie: higher surrogate precision rarely tunes better","Surrogate accuracy doesn't guarantee better config tuning","Large-scale study: model accuracy often misleads tuners","Why accurate surrogate models still fail at tuning","97% of the time, surrogate accuracy fails to aid tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3440,"prompt_tokens":1152,"completion_tokens":2288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":768,"completion_tokens_details":{"reasoning_tokens":2211}},"tokens_in":768,"tokens_out":2288,"duration_ms":17011,"temperature":1.0,"reasoning_tokens":2211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:18:19.128385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute model accuracy on the subset of test configurations closest to the true optimum (for example within a small Hamming distance) or on the top-ranked predicted configurations: if this near-optimum accuracy correlates strongly and positively (Spearman $\\rho > 0.7$) with the achieved tuning quality for a majority of the 29 systems, then the paper's conclusion would be shown to be an artifact of whole-space aggregate accuracy rather than a failure of accuracy as such.","supporting_citations":[{"cited_title":"Bestconfig: tapping the performance potential of systems via automatic configuration tuning,","cited_arxiv_id":null,"evidence_quote":"Its quoted criticism of model inaccuracy motivates the belief and supplies the BestConfig model-free baseline."},{"cited_title":"A comprehensive survey on fitness landscape analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the fitness landscape analysis concepts (guidance, ruggedness) used to explain why accuracy fails to predict tuning quality."}],"review_version":1}