{"id":"2487c088-638d-41ed-9556-058af1702be4","arxiv_id":"2507.01638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A tailored SHAP-based analysis of rmnk-landscapes shows that the ruggedness parameter k drives multi-objective algorithm performance and that PLOS-net, C-PLOS-net, and funnel feature sets are complementary.","lead":"This paper asks which properties of random test landscapes best predict how well three multi-objective optimization algorithms find trade-off solutions. It finds that the number of interactions between variables, a ruggedness measure, dominates algorithm difficulty, and that different landscape features matter for different algorithms and landscapes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that k is the most influential benchmark parameter rests on SHAP meta-representations from an unvalidated surrogate model; the paper reports no predictive accuracy and the 18-instance visual inspection cannot distinguish true k-hardness from surrogate artifacts.","rationale":"The reader's weakest_assumption identifies exactly the surrogate-model validity issue, and my independent reading of Section 4, Section 5.1.1, and Section 6 lands on the same point. The paper has real strengths: the C-PLOS-net and funnel features have independent grounding in prior work, the pipeline is standard, and the code and data are released. The concern is not that the methodology is wrong, but that the paper's central quantitative claim is supported only by an unquantified surrogate and a small, visually interpreted test set. This is load-bearing because every downstream conclusion—the k-hardness ordering, the per-landscape feature importance, and the complementarity of feature sets—is expressed via SHAP values of that surrogate. A finite-sample or bias issue in the surrogate would not just weaken a numeric coefficient; it would change the headline scientific claim. I therefore agree with the CONDITIONAL verdict. I do not recommend ACCEPT without the requested ground-truth validation, and I do not recommend REJECT because the concern is readily checkable, the data are public, and the methodology is standard. The concrete test above is one specific check that would settle whether the concern lands: it separates surrogate-driven conclusions from direct empirical ones and quantifies the error that the paper currently only asserts to be minor.","tokens_in":11649,"tokens_out":1781,"duration_ms":18958,"concrete_test":"Re-run the Section 5.1.1 analysis replacing SHAP values with direct ground-truth performance data: (1) compute the actual reso of PLS, GSEMO, and NSGA-II on the 18 test instances; (2) report MAE and R2 of the final RF-MTR model on those 18 test instances, separately for each algorithm and for each k value; and (3) test whether the hardness ordering across k (k=4 worse than k=2 worse than k=1) holds directly on ground-truth reso with a paired significance test across the 18 instance-parameter cells. If the ground-truth ordering by k matches the surrogate-based ordering and the model's per-k MAE is small relative to the k-group performance gaps, the central claim stands; otherwise the k-dominance conclusion is an artifact of the surrogate.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing step is Section 5.1.1: the claim that k (number of interactions) is the most influential benchmark parameter is derived from visual inspection of Figure 2, which plots PCA projections of SHAP meta-representations for 18 test instances. Those SHAP values come from the MTR Random Forest model. Section 4 says the final model was selected by lowest cross-validation error, and Section 6 says predictions 'despite minor errors, closely approximate ground-truth reso performance,' but no MAE, R2, or other accuracy number appears in the paper. Two concrete risks follow. First, if the surrogate is systematically biased with respect to k, the apparent k-driven separation in the embedding could be a property of the surrogate, not of true algorithm performance. Second, SHAP values for tree ensembles are local explanations of model predictions; they cannot separate 'feature genuinely drives hardness' from 'feature drives the model's prediction error.' The paper nowhere validates that SHAP importance patterns on held-out instances match feature-performance relationships measured directly on ground-truth performance data. The k-dominance conclusion may be correct, but the evidence presented—18 test instances visualized after PCA—is not sufficient to establish it; no error bars, no direct ground-truth-reso versus k analysis, and no ablation showing the conclusion is robust to surrogate accuracy. The Section 7 claim that combining two feature sets is essential inherits the same weakness: the 'essential combination' is identified from SHAP contributions of the surrogate, so surrogate error would propagate directly into the conclusions about which landscape feature combinations matter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an 'algorithm footprint' analysis pipeline for multi-objective combinatorial optimization. It trains a multi-target regression (MTR) Random Forest to predict the performance of PLS, GSEMO, and NSGA-II on ρMNK-landscapes from 41 landscape features (24 PLOS-net, 10 C-PLOS-net, and 7 funnel features, including two newly proposed ones). SHAP values provide local feature importances, which are clustered hierarchically to form performance regions and algorithm footprints. The central empirical claims are that the number of interactions k is the most influential benchmark parameter for problem hardness, and that combining feature sets (PLOS-net, C-PLOS-net, funnel) is essential for understanding algorithm-specific behavior. The hypervolume analysis is mentioned but deferred to a repository.","tokens_in":11900,"tokens_out":4618,"duration_ms":42963,"significance":"If the findings are valid, the paper contributes a per-instance, per-algorithm tool for comparing MOO algorithms and a concrete observation about ρMNK-landscape hardness, which would be of interest for benchmark design and algorithm selection. The data and code are publicly available, which is a reproducible strength. However, the central conclusions rest on SHAP values from a surrogate model whose predictive accuracy is never numerically reported, and on visual patterns in only 18 test instances. The manuscript is clearly written and the methodology is standard, but the evidence for the headline claims is currently incomplete. With added validation of the surrogate and direct ground-truth checks, the contribution would be substantial for the meta-learning and landscape analysis community.","major_comments":[{"comment":"The manuscript states in Section 4 that the final MTR model is evaluated on the test set using Mean Absolute Error (MAE) and R-squared (R2), and in Section 6 that 'Algorithm footprints are based on MTR model predictions, which, despite minor errors, closely approximate ground-truth reso performance,' but no numeric values for MAE or R2 appear anywhere in the paper. Since every subsequent conclusion in Section 5 is derived from SHAP values of this surrogate model, the absence of any measure of predictive accuracy makes it impossible to judge whether the footprint patterns reflect true algorithm performance or artifacts of the model's errors. The authors should report the test-set and cross-validation MAE/R2 for the selected RF model and for the compared baselines, so that the central claim can be assessed.","section":"Section 4 and Section 6"},{"comment":"The conclusion that 'the k benchmark parameter has the most influence on the hardness of the problem instances for the three algorithms' is based on visual inspection of PCA-projected SHAP meta-representations of only 18 test instances. This is a qualitative pattern, not a quantitative analysis: the paper provides no separation measure, no error bars, and no statistical test linking k to ground-truth reso. Moreover, SHAP values explain the model's predictions, not necessarily ground-truth performance; a systematic bias of the surrogate with respect to k would produce exactly the observed separation. To make the claim load-bearing, the authors should add a direct analysis of ground-truth reso as a function of k (e.g., an ANOVA or a per-instance plot with variability), and/or validate that the SHAP importance patterns on held-out instances match feature-performance relationships computed directly on ground-truth data.","section":"Section 5.1.1, Figure 2"},{"comment":"The conclusion that 'combining two sets of features is essential' is not supported by the experimental design reported in Section 4. The Sequential Forward Feature Selection (SFFS) procedure operates on the union of all 41 features and selects a single 23-feature subset; there is no comparison between models trained on individual feature groups (PLOS-net, C-PLOS-net, funnel) and models trained on their combinations. The observed importance of features from different groups in the SHAP decision plots is an indirect indication, not a test of complementarity. The authors should either include an ablation study comparing feature-group combinations, or temper the claim to state that the analysis identifies useful features within the combined set without asserting that the combination is essential.","section":"Section 7 and Section 4"}],"minor_comments":[{"comment":"In the PLS footprint description, 'a high number of objectives (k = 3)' should read 'm = 3', and later 'k∈{2,3}' should read 'k∈{2,4}', since k takes values 1, 2, 4 in Table 1.","section":"Section 5.1.1"},{"comment":"The acronym 'MTEL' for Multi-Task Elastic Net is inconsistent with the earlier 'MTEN' in the same section; please unify the notation.","section":"Section 4"},{"comment":"The abstract and introduction state that the analysis uses both resolution and hypervolume metrics, but Section 6 says the hypervolume results are omitted due to space and are only in the repository; please clarify the scope in the abstract or include a summary of the hv results.","section":"Section 6 and Abstract"},{"comment":"There are several typographical errors that should be corrected: 'prepossessing' for 'preprocessing' in Section 4, 'Regulrization' in the title of reference [41], and 'the the benchmark suite' in Section 2.2.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the methodology is appropriate, but the central claim about k-dominance depends entirely on an unvalidated surrogate model. The missing MAE/R2 is simple to add and would substantially de-risk the contribution. I also encourage the authors to run a direct ground-truth k-versus-performance analysis, since the current visual evidence is not sufficiently convincing on its own."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a competent, mostly incremental application of the authors' own algorithm-footprint pipeline to rmnk-landscapes, with two genuinely new funnel features and a first per-landscape/per-algorithm feature-importance analysis. Second, the paper's headline—k as the dominant hardness driver—is asserted from visual inspection of SHAP plots over 18 test instances, and the surrogate model's accuracy is never reported. The claim may well be right; the evidence as written is not enough to carry it.\n\nWhat's new and good: pos_num and pos_strength are new, defined independently of algorithm performance, and the code and data are released. The train/test split is clean, SFFS and hyperparameter tuning are standard, and the clustering/footprint analysis is a reasonable way to look for problem/algorithm interactions. The paper is honest that hv results are in the repository. No circularity: features are computed from full enumeration, not from algorithm runs.\n\nSoft spots, in proportion. The missing MAE/R2 is the biggest one. Section 4 says the final model is evaluated on the test set with MAE and R2; Section 6 says predictions 'despite minor errors' approximate ground truth; no numbers appear. Without them, SHAP-based feature importance and the k conclusion are only as trustworthy as an unvalidated surrogate. SHAP tells you what the model uses, not directly what the landscape does to the algorithms. A simple scatter of ground-truth reso vs. k for the 18 test instances would go a long way. Also, 18 instances—one per parameter combination, with no error bars—is thin for a claim about 'the most influential benchmark parameter,' even if the pattern looks clear in the PCA plots. The qualitative reading of SHAP decision plots in Figures 4–6 is also a bit freewheeling; it is plausible but not tightly argued.\n\nNone of this is fatal. The pipeline is coherent, the code is available, and the feature-importance decompositions could be useful to people working on MOO benchmark design. The fix is reporting: add test-set accuracy, add a ground-truth check on k, and show that the cluster-level feature importances are stable across the 10 instances per parameter combination rather than one test instance each.\n\nWho it's for: researchers in multi-objective landscape analysis and automated algorithm selection on rmnk-landscapes. It deserves a serious referee; I'd send it out with a request for the missing validation, not desk reject.","headline":"Competent, incremental footprint analysis whose central k-hardness claim is under-supported by missing surrogate accuracy and a thin 18-instance visual check.","tokens_in":12491,"tokens_out":3814,"would_cite":false,"duration_ms":45672,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the number of interactions $k$ dominates how hard $\\rho$mnk-landscapes are for PLS, GSEMO, and NSGA-II, and that the PLOS-net, C-PLOS-net, and funnel feature sets must be combined to explain why.","keywords":["landscape analysis","multi-objective combinatorial optimization","algorithm footprints","feature importance","SHAP","ρmnk-landscapes","C-PLOS-net"],"falsifier":"Report the test-set R-squared and mean absolute error of the multi-target regression model, or replace the random forest with a linear or neural surrogate and rerun the SHAP footprint pipeline; if the footprints change substantially, or if the surrogate explains little variance, then the claimed dominance of $k$ and the per-cluster feature combinations are artifacts of the model rather than properties of the landscapes and algorithms.","tokens_in":11432,"feed_emoji":"📈","tokens_out":10744,"duration_ms":109197,"temperature":0.7,"pith_summary":"This paper asks which landscape features make multi-objective combinatorial optimization problems hard for specific algorithms. Working with small $\\rho$mnk-landscapes (binary strings of length 16, two or three objectives, interaction counts $k\\in\\{1,2,4\\}$, objective correlations $\\rho\\in\\{-0.4,0,0.4\\}$), it trains a multi-target regression model to predict the resolution and hypervolume achieved by Pareto Local Search, GSEMO, and NSGA-II from 41 landscape features. It then converts the model's predictions into per-instance, per-algorithm feature-importance vectors with SHAP, clusters those vectors, and reads off which feature combinations drive performance in each performance region. The central claim is that the number of interactions is the most influential benchmark parameter: $k=1$ instances are easy and $k=4$ instances are hard, while objective count and correlation play secondary roles. The paper also claims that the PLOS-net, C-PLOS-net, and funnel feature groups are complementary and that their combination, not any single group, is what explains performance on particular landscape-algorithm pairs.","feed_headline":"Interaction count, not objective count, decides algorithm hardness","feed_subtitle":"The number of interacting variables outweighs objective correlation and dimension in setting problem difficulty.","key_machinery":"The carrying mechanism is the algorithm-footprint pipeline. A multi-target regression model predicts the performance of all three algorithms from one 41-dimensional feature vector per instance; SHAP values turn the trained model into a per-instance, per-algorithm vector of local feature importances (a meta-representation); hierarchical clustering groups these meta-representations into 14 performance regions; and the aggregated feature importances per region define each algorithm's footprint. The features themselves come from three named network models of the search space: PLOS-net, its compressed form C-PLOS-net, and funnel metrics based on non-dominated sorting ranks. The pipeline is what allows landscape features to be tied to algorithm behavior locally rather than through a single global importance ranking.","core_discovery":"The paper's central claim is that, on this benchmark suite, the number of interactions $k$ is the dominant determinant of instance hardness for all three algorithms under the resolution metric. Instances with $k=1$ sit in the high-performance region of the footprint plots and instances with $k=4$ in the low-performance region, independent of objective correlation and objective count. A second claim is that the three landscape-feature families—the 24 PLOS-net features, the 10 C-PLOS-net features, and the seven funnel features—carry complementary information: the best predictive model uses a selected subset drawn from all three, and the combined set is what becomes significant for specific landscapes. A third claim is that different algorithms reach similar performance on the same landscape for different reasons, visible as distinct clusters of SHAP feature importance, and that two newly added funnel features, pos_num and pos_strength, appear among the top-ranked features across nearly all landscapes.","pith_inferences":["If the dominance of $k$ transfers beyond $n=16$, algorithm selection for $\\rho$mnk-landscapes could be stratified by interaction density before expensive network features are computed, and benchmark generators could use $k$ as the primary hardness axis.","The paper notes that hypervolume-based footprints produce different top feature sets and less algorithm complementarity; a direct comparison of resolution and hypervolume footprints on the same instances could show whether the 'k dominates' claim is metric-dependent.","Because the footprints come from one surrogate model, an ablation that trains the same SHAP pipeline on each feature group in isolation, or on permuted labels, would test how much of the reported complementarity is a property of the landscapes rather than of the random forest.","The same footprint machinery could be applied to larger $\\rho$mnk instances or other combinatorial structures such as multi-objective QAP instances, where the paper itself notes data collection is the main barrier."],"forward_implications":["For the resolution metric, instance hardness on $\\rho$mnk-landscapes is set mainly by $k$: $k=1$ instances are easy across the portfolio, $k=4$ instances hard, and $m$ with $\\rho$ play secondary roles.","A single feature family (PLOS-net, C-PLOS-net, or funnel features alone) is not enough; the best predictive model combines all three families, so future feature engineering for this benchmark family should treat them as complements.","The two new funnel features, pos_num and pos_strength, are consistently top-ranked in nearly all landscapes, making them candidate cheap indicators of hard instances.","PLS, GSEMO, and NSGA-II have genuinely different footprints: PLS excels on negatively correlated three-objective landscapes with few interactions while NSGA-II struggles there, and the ordering reverses on two-objective landscapes, so algorithm portfolios benefit from all three."],"supporting_citations":[{"why":"Supplies the 24 base PLOS-net landscape features that describe Pareto local optimal solutions networks.","marker":"[24]"},{"why":"Introduces the compressed C-PLOS-net model and its 10 additional features, and supplies the algorithm performance data reused here.","marker":"[25]"},{"why":"Provides the funnel features derived from non-dominated sorting ranks that capture global landscape structure.","marker":"[31]"},{"why":"Defines the algorithm-footprint methodology (multi-target regression, SHAP, clustering) that the paper adapts to multi-objective optimization.","marker":"[28]"},{"why":"Defines rmnk-landscapes with correlated objectives, the benchmark generator that determines the instances' parameters k, m, and rho.","marker":"[38]"},{"why":"Referenced for the SHAP explanation method used to compute local feature importance from the trained model.","marker":"[37]"},{"why":"Defines Pareto Local Search and its termination at a Pareto local optimum set, one of the three algorithms footprinted.","marker":"[32]"},{"why":"Introduces NSGA-II, the second algorithm whose performance the meta-model predicts.","marker":"[10]"},{"why":"Introduces GSEMO, the third algorithm in the portfolio.","marker":"[21]"}],"fun_headline_variants":["Interaction count, not objective count, decides hardness","k interactions trump objectives for algorithm difficulty","Hardness driven by k: interactions outweigh objectives","For rmnk landscapes, k predicts performance, not objectives","More interactions, harder problems: k is the key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the trained prediction model being accurate enough that the SHAP importance scores it produces reflect how the algorithms really behave, but the paper asserts this accuracy is good without reporting the model's numerical error.","fun_headline_variants_meta":{"raw":{"variants":["Interaction count, not objective count, decides hardness","k interactions trump objectives for algorithm difficulty","Hardness driven by k: interactions outweigh objectives","For rmnk landscapes, k predicts performance, not objectives","More interactions, harder problems: k is the key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1372,"prompt_tokens":852,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":468,"tokens_out":520,"duration_ms":6240,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:46:38.537191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Report the test-set R-squared and mean absolute error of the multi-target regression model, or replace the random forest with a linear or neural surrogate and rerun the SHAP footprint pipeline; if the footprints change substantially, or if the surrogate explains little variance, then the claimed dominance of $k$ and the per-cluster feature combinations are artifacts of the model rather than properties of the landscapes and algorithms.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 24 base PLOS-net landscape features that describe Pareto local optimal solutions networks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the compressed C-PLOS-net model and its 10 additional features, and supplies the algorithm performance data reused here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the funnel features derived from non-dominated sorting ranks that capture global landscape structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the algorithm-footprint methodology (multi-target regression, SHAP, clustering) that the paper adapts to multi-objective optimization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines rmnk-landscapes with correlated objectives, the benchmark generator that determines the instances' parameters k, m, and rho."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Referenced for the SHAP explanation method used to compute local feature importance from the trained model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Pareto Local Search and its termination at a Pareto local optimum set, one of the three algorithms footprinted."},{"cited_title":"Meyarivan","cited_arxiv_id":null,"evidence_quote":"Introduces NSGA-II, the second algorithm whose performance the meta-model predicts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces GSEMO, the third algorithm in the portfolio."}],"review_version":1}