{"id":"381b411a-a4e2-4289-aad5-16cc661892a7","arxiv_id":"2505.07750","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Non-informative features and a scale-only feature achieve strong prediction errors under common algorithm-selection evaluation schemes, without actually selecting better algorithms.","lead":"This paper shows that two common evaluation practices for algorithm selection meta-models can produce misleadingly good results. It demonstrates that leave-instance-out evaluation rewards irrelevant features, and that scale-sensitive performance metrics inflate apparent model quality without improving algorithm ranking.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LIO demonstration depends on within-class rank similarity that is only evidenced by the authors' own class-feature experiment and a GitHub reference; its generality across dimensions and portfolios is unverified.","rationale":"I read the paper as a methodological caution rather than a proof of universal failure. Both central claims are worded as existence claims ('can result', 'can produce'), and the experiments do provide concrete demonstrations: a single class-identity feature yields PRE 0.05 under LIO but fails under LPO, and a single scale feature drastically reduces MSE on target precision without improving ranking. Those demonstrations are internally coherent and reproducible via the provided repository, which counts as real support. The load-bearing weakness is the step from these demonstrations to broad recommendations about community practice. For the LIO argument, the key mechanism is that class identity predicts algorithm rankings because instances within a class are similar. The paper calls this mild and cites its own GitHub for support; the class-feature experiment itself is both the evidence and the phenomenon. That is not a fatal circularity for a 'can happen' claim, but it does mean the generality of the flaw across dimensions, instance counts, portfolios, and benchmark suites is unestablished. This matches the reader's identified weakest assumption and conditional verdict. I am not moving the verdict because the paper's stated central claims are supported in the tested setting; the condition is to temper the generalizing conclusions or add evidence across configurations.","tokens_in":13748,"tokens_out":5880,"duration_ms":73670,"concrete_test":"Re-run the Section 2 class-feature LIO experiment on COCO in 10D and 20D, with the same five pymoo solvers and 15 instances per class, and additionally compute the median pairwise Kendall tau between algorithm-rank vectors for instances within the same class versus across classes. If the class-feature PRE rises toward the mean-model baseline, or if within-class rank agreement is no larger than between-class agreement, then the demonstrated LIO flaw is specific to the 5D/default-portfolio configuration and the general recommendation should be narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central LIO claim is conditional on a premise that the paper itself flags as 'a mild assumption': instances within a COCO problem class have similar algorithm rankings. The class-feature meta-model achieving PRE 0.05 is the only in-paper evidence for this premise, which makes the demonstration partly circular: the experiment simultaneously assumes and illustrates the effect. The cited support [22] is the authors' own GitHub repository, not independent evidence, and the cited external references [16, 26, 28] concern landscape properties and instance similarity, not specifically rank similarity for this portfolio. The experiments are confined to 5D, 15 instances per class, five default-configured pymoo solvers, and one benchmark. If within-class rank similarity weakens in higher dimensions, with more instances, or with a more diverse portfolio, the class/lookup mechanism would no longer drive PRE down to 0.05, and LIO would not be misleading in that setting. Because the conclusion recommends broad community-practice changes, this scope limitation is load-bearing rather than cosmetic. The scale-sensitive metric claim is more robust logically: showing that MSE gains from a scale feature do not translate to ranking gains is a valid counterexample. Its external scope is similarly limited, but the core mechanism does not depend on a contested similarity assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that two common evaluation practices in algorithm selection (AS) for continuous black-box optimization are flawed. First, it claims that the leave-instance-out (LIO) evaluation on the COCO benchmark can reward non-informative features and meta-models through spurious correlations with problem class, producing over-optimistic results. This is demonstrated with three feature sets (ELA, hand-crafted non-informative statistics, and a single problem-class indicator) and five pymoo solvers on 5D COCO instances, where the class-only meta-model reaches PRE ≈ 0.05 under LIO but not under leave-problem-out (LPO). Second, it claims that scale-sensitive performance metrics, such as target precision, can mislead meta-model evaluation: a model trained with a single scale-correlated feature appears to outperform a mean baseline on MSE, but this advantage vanishes when the predictions are converted to rankings and evaluated with pairwise ranking error. The paper concludes with recommendations to avoid LIO and scale-sensitive metrics in AS evaluation.","tokens_in":13908,"tokens_out":3751,"duration_ms":40080,"significance":"If the claims hold, the paper makes a useful contribution by explicitly demonstrating two evaluation pitfalls that are likely widespread in the continuous black-box optimization AS community. The experiments are carefully constructed and the code is available, which supports reproducibility. The section on scale-sensitive metrics is particularly effective: the counterexample where a single scale feature yields an order-of-magnitude MSE improvement with no ranking improvement cleanly illustrates why scale-sensitive metric evaluation can be misleading. The LIO counterexample is also thought-provoking, although its strength depends on a similarity premise that is not independently established. The paper is appropriately cautious in places (e.g., noting that it cannot generally test feature informativeness), but the scope of the headline claims is broader than the supporting experiments.","major_comments":[{"comment":"The label 'non-informative features' is not justified. These features are aggregates of fitness values (e.g., mean, median, standard deviation, quantiles) under random scalar and transformation choices. Such statistics can carry information about the scale, spread, or difficulty of the objective function, and may therefore be genuinely informative for algorithm performance rather than only for class identity. The paper should either rename these features as 'not intentionally designed for AS' or provide a control showing that they do not correlate with algorithm performance in a setting where class identity is removed (e.g., within-class regression or LPO with multiple classes).","section":"Section 2, paragraph before cases (a)-(d)"},{"comment":"The LIO demonstration is limited to 5D problems, 15 instances per class, five default-configured pymoo solvers, and a single benchmark. The conclusion that the LIO methodology is flawed and should be avoided is stated broadly, but the experiments do not show whether the spurious-success effect persists in higher dimensions, with more instances per class, or with portfolios that have more varied behavior within a class. Since the paper recommends broad community-practice changes, this scope limitation is load-bearing rather than cosmetic. The paper should either add experiments in other settings or explicitly restrict the claim to the demonstrated configuration.","section":"Section 2.2, Figure 1"},{"comment":"The scale-sensitive metric argument is sound as a counterexample, but the final recommendation that 'metrics like PRE or classification accuracy are more suitable' may be too strong. Ranking metrics discard the magnitude of performance differences, which can be important when some algorithm pairs are nearly tied or when the cost of a wrong selection depends on the gap. The paper should temper this recommendation or discuss situations where scale-aware metrics are still needed and how they should be normalized.","section":"Section 3.3, recommendation bullets"}],"minor_comments":[{"comment":"The text says each algorithm is run 30 times per instance to obtain mean rank, but the definition of PRE in Eq. (2) refers to rankings 'established through 100 runs'. Please align these numbers.","section":"Section 2.1, performance measurement"},{"comment":"The sentence 'the second uses the so-called non-informative features. These features are designed to identify the class of a problem' is confusing: the non-informative features are not designed to identify class; the class feature is. Rephrase to avoid implying that the random-transform features were built as class indicators.","section":"Section 2.1, feature sets"},{"comment":"The notation in the summation is garbled: it should be a sum over ordered pairs (a_i, a_j) in A^2 with a_i ≠ a_j, rather than 'A 2, a_i ≠ a_j'. Please fix the typesetting.","section":"Equation (1)"},{"comment":"Reference [34] appears to duplicate reference [28] (same paper). Please consolidate.","section":"References"},{"comment":"The left panel uses a logarithmic y-axis for MSE; consider stating this explicitly in the caption, and note that the Wilcoxon test is applied to 24 paired points.","section":"Section 3.2, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a methodologically important position piece, but its central LIO claim relies on a similarity assumption that is supported mainly by the authors' own repository and by the very experiment that assumes it. I would like to see either independent evidence for within-class rank similarity or a clearly delimited claim. The scale-sensitive metric section is strong and could stand as the main contribution if the LIO part is revised. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper has a clean, controlled demonstration that under leave-instance-out evaluation on COCO, a single feature carrying only the problem class achieves PRE 0.05 — better than ELA features — while under leave-problem-out it collapses to chance. That is a stark and convincing illustration of a real methodological hazard. Second, it shows that predicting a scale-sensitive target (target precision) with a scale feature can improve MSE by orders of magnitude while giving zero improvement on ranking error. That is a clean counterexample to the practice of reporting prediction error on scale-sensitive targets as evidence of AS quality.\n\nThe paper does some things well. It explicitly defines scale-sensitive vs scale-independent metrics for AS, which I don't recall being done before. It constructs deliberately minimal 'non-informative' features rather than relying on accidental correlations. It acknowledges that its methodology cannot be used to certify that a feature set is non-informative, and it resists naming specific papers, which keeps the discussion useful. The code is available. The statistical tests are appropriate.\n\nWhere it is soft: the experiments are 5D only, 15 instances per class, five default-configured pymoo solvers, one benchmark. The LIO result depends on within-class rank similarity. In this setting, the class-feature result itself shows the premise holds, so it's not circular, but the generality is untested. If within-class rank similarity weakens in higher dimensions or with a more diverse portfolio, LIO would be less problematic. The paper nonetheless recommends broad community-practice changes. The LIO critique is really about using LIO to claim generalization to new problem classes; for selecting among algorithms on instances of known classes, LIO is a legitimate evaluation. The conclusion should say that, not 'LIO is flawed.'\n\nThe scale-sensitive section is more robust. The disconnect between MSE gains on target precision and zero PRE gains is a solid logical argument, and the point about baselines being scale-agnostic is well taken. Minor: the 'non-informative' features are fitness-value statistics and may carry real difficulty information; the class feature is the load-bearing evidence, and the paper should make that explicit.\n\nOverall: this is a useful methodological warning, honest in tone, with clean demonstrations. It deserves a serious referee. I would recommend accept after revision that tempers the generalizing language and clarifies when LIO is and isn't appropriate.","headline":"A clean, well-scoped demonstration of two pitfalls in algorithm selection evaluation that deserves review but overclaims its generality.","tokens_in":14464,"tokens_out":4374,"would_cite":true,"duration_ms":43381,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two common benchmarking habits can make algorithm-selection results look far better than they really are.","keywords":["algorithm selection","leave-instance-out evaluation","leave-problem-out evaluation","spurious correlations","scale-sensitive metrics","target precision","continuous black-box optimization","benchmarking pitfalls"],"falsifier":"Construct, or find, a benchmark whose instances within a class are deliberately made as dissimilar in algorithm ranking as instances across classes; if a class-identity meta-model still achieves near-0.05 pairwise ranking error under leave-instance-out evaluation, the paper's spurious-correlation explanation would be wrong, whereas if the advantage vanishes, the mechanism is confirmed.","tokens_in":13501,"feed_emoji":"⚠️","tokens_out":8816,"duration_ms":83161,"temperature":0.7,"pith_summary":"Algorithm selection in continuous black-box optimization is usually evaluated by training a meta-model on landscape features and checking how well it chooses an optimizer. The paper argues that two common evaluation choices can inflate those results: leave-instance-out testing on the COCO benchmark, where instances within the same problem class are similar enough that a feature encoding only the class identity predicts held-out performance; and scale-sensitive performance targets such as target precision, where a model can shrink its error by learning the problem's scale rather than which algorithm is better. Under leave-instance-out, deliberately non-informative features and a single class-identity feature both beat sensible baselines, while under leave-problem-out the advantage disappears. Under scale-sensitive targets, a meta-model using one scale-related feature beats the mean-precision baseline by an order of magnitude, yet its rank-based error is no better than the baseline. The authors conclude that such evaluation flaws can invalidate core claims in the field and should be replaced or supplemented by rank-based, scale-independent checks.","feed_headline":"Leave-instance-out tests can make useless features look great","feed_subtitle":"Two evaluation choices can make algorithm selectors look better than they are, and simple checks reveal why.","key_machinery":"The first load-bearing mechanism is the combination of a benchmark with 24 problem classes and several instances per class, a leave-instance-out (LIO) split in which instances of every class appear in both training and test sets, and Pairwise Ranking Error (PRE), the fraction of algorithm pairs whose predicted order disagrees with the true order. Because instances within a class resemble one another, class identity is a shortcut: a meta-model can lower PRE by memorizing which class an instance belongs to. The second mechanism is target precision, $f(x_{\\mathrm{best}})-f(x_{\\mathrm{opt}})$, which the paper defines as scale-sensitive because rescaling the objective function changes its value; the feature $f_{\\mathrm{scale}}=\\max(Y)-\\min(Y)$ then lets a regressor reduce MSE simply by tracking scale. The paper constructs non-informative features by applying random aggregate, transformation, and scalar functions to sampled fitness values, and uses the class label as a worst-case feature.","core_discovery":"The authors claim that 'leave-instance-out' (LIO) evaluation is flawed in the COCO setting: because instances of the same problem class resemble each other in both landscape and algorithm performance, a meta-model can achieve low pairwise ranking error by learning which class an instance belongs to rather than by learning anything useful for selecting an algorithm. They support this with deliberately non-informative features built only from summary statistics of sampled function values, and with a worst-case feature that is simply the problem class; under LIO both reach PRE around 0.13 and 0.05 respectively, comparable to or better than exploratory landscape analysis features, while under leave-problem-out all meta-models perform similarly. They further claim that scale-sensitive performance metrics, specifically target precision $f(x_{\\mathrm{best}})-f(x_{\\mathrm{opt}})$, conflate problem scale with algorithm quality: a random-forest meta-model using only the feature $\\max(Y)-\\min(Y)$ beats a mean-precision baseline by an order of magnitude in MSE under leave-problem-out, but when predictions are converted to ranks, its pairwise ranking error is indistinguishable from the baseline. The conclusion is that these evaluation choices create spurious success, not genuine algorithm-selection ability.","pith_inferences":["The same leave-instance-out critique should apply to any performance-prediction task on benchmarks with near-duplicate instance families, such as runtime or difficulty prediction for shifted, scaled, and rotated problem instances.","A natural extension is to normalize objective values before computing target precision, or to use a relative gap; the paper's logic predicts that the apparent advantage of scale-aware models would then shrink or disappear.","A testable consequence is that a new feature set whose leave-instance-out performance lies between the mean baseline and the class-identity baseline cannot be distinguished from spurious correlation without a stronger evaluation protocol.","The scale-sensitivity argument suggests that meta-model results across papers are not comparable unless the scale distribution of the benchmark problems is matched or controlled."],"forward_implications":["A strong result obtained only under leave-instance-out evaluation on a benchmark whose instances cluster by problem class is not evidence of generalization; class memorization can reproduce it.","A meta-model that predicts target precision well, as measured by MSE or MAE, can fail entirely at the actual task of ranking algorithms, so rank-based metrics such as PRE should accompany any scale-sensitive regression target.","A baseline that predicts mean target precision is too weak when problems differ in scale; accounting for scale is necessary before a precision-prediction improvement means anything.","Feature-importance and explainability analyses on such meta-models are unreliable, because they will inflate the importance of class-correlated and scale-sensitive features.","Reporting leave-problem-out evaluation, or at least a permutation or class-label check, would expose the spurious success that LIO hides."],"supporting_citations":[{"why":"Supplies the benchmark suite whose 24 problem classes and multiple instances per class create the within-class similarity that drives the LIO demonstration.","marker":"[7]"},{"why":"Supports the claim that instances within the same problem class have similar landscape properties and similar algorithm performance.","marker":"[16]"},{"why":"Companion code for the experiments and the basis for the mild assumption that algorithm performance is similar within a class.","marker":"[22]"},{"why":"Provides the ELA feature set used as the reference feature-based meta-model in the LIO experiments.","marker":"[24]"},{"why":"Defines target precision and fixed-budget performance regression, the scale-sensitive target the paper critiques.","marker":"[11]"},{"why":"Documents the LIO and LPO evaluation strategies and the algorithm-selection meta-model setup that the paper argues are used inconsistently.","marker":"[31]"},{"why":"Supplies the husky-versus-wolf example of spurious correlations used to motivate why class-memorization can look like success.","marker":"[27]"},{"why":"Establishes that many ELA features are scale-sensitive, showing the f_scale critique reaches existing feature sets.","marker":"[34]"}],"fun_headline_variants":["Instance leakage and scale bias overstate selector performance","Leave-instance-out tests can mask selector failure","Scale-sensitive metrics inflate algorithm selection accuracy","Benchmark pitfalls: non-informative features score high"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstration that leave-instance-out evaluation rewards useless features rests on the assumption that COCO instances within the same problem class are similar enough, in landscape and in algorithm performance, that a feature encoding only the class identity can predict held-out performance; if within-class instances were as diverse as across-class instances, the class-only feature would not achieve its low error.","fun_headline_variants_meta":{"raw":{"variants":["Instance leakage and scale bias overstate selector performance","Leave-instance-out tests can mask selector failure","Scale-sensitive metrics inflate algorithm selection accuracy","Benchmark pitfalls: non-informative features score high"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2890,"prompt_tokens":1000,"completion_tokens":1890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1830}},"tokens_in":616,"tokens_out":1890,"duration_ms":15359,"temperature":1.0,"reasoning_tokens":1830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:07:54.149910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct, or find, a benchmark whose instances within a class are deliberately made as dissimilar in algorithm ranking as instances across classes; if a class-identity meta-model still achieves near-0.05 pairwise ranking error under leave-instance-out evaluation, the paper's spurious-correlation explanation would be wrong, whereas if the advantage vanishes, the mechanism is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark suite whose 24 problem classes and multiple instances per class create the within-class similarity that drives the LIO demonstration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that instances within the same problem class have similar landscape properties and similar algorithm performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Companion code for the experiments and the basis for the mild assumption that algorithm performance is similar within a class."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ELA feature set used as the reference feature-based meta-model in the LIO experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines target precision and fixed-budget performance regression, the scale-sensitive target the paper critiques."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the LIO and LPO evaluation strategies and the algorithm-selection meta-model setup that the paper argues are used inconsistently."}],"review_version":1}