{"id":"dabb2a90-52d3-4a8d-8f74-9860ac15c1c2","arxiv_id":"2411.17999","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-indicator ranking method sorts per-run performance scores with Pareto dominance and uses four counting rules to rank many-objective optimizers.","lead":"This study proposes ranking multi-objective optimization algorithms by treating several performance metrics as competing objectives and grouping per-run scores into Pareto dominance levels. The approach lets researchers combine all quality metrics into a single comparison, which matters because no single indicator captures every aspect of optimizer performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section V.D sums Pareto-level counts across problems/objective counts without justifying that level l is equivalent across NDS runs, so the overall Table V ranking and the 'regardless of objectives' claim rest on an unproven commensurability assumption.","rationale":"The paper's single-problem pipeline is clear and internally consistent, and the derivations of the four scoring schemes are straightforward. The main gap is the unargued aggregation of level counts across heterogeneous problems and objective counts. This is the most load-bearing assumption because the headline claim is about global reliability; without it, the overall ranking is arbitrary. The proposed Borda-count check would settle whether the aggregation is benign in practice. I agree with the reader's conditional verdict: the method is plausible but the strong claim in Section I needs either a commensurability argument or a restriction to single-problem settings. I would not reject the paper outright because the per-problem use case remains potentially useful.","tokens_in":17753,"tokens_out":6776,"duration_ms":62426,"concrete_test":"Recompute the overall rankings using a Borda-count baseline: for each of the 45 (problem, M) combinations, apply the four proposed techniques to that problem's level-count table to obtain a rank for each algorithm; then aggregate these per-problem ranks by average rank. Compare this aggregated ranking with Table V. If the top-ranked algorithm changes or the Kendall tau correlation with Table V is below 0.8, the Section V.D aggregation assumption is consequential and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.D (and Section V.C) forms the overall ranking by summing, for each algorithm, the number of points per Pareto level across all 15 MaF problems and across M = 5, 10, and 15 objective counts, producing Table IV and then scoring with Eqs. (17), (18), and (21). This requires that a point on level l in one problem's NDS is comparable to a point on level l in another problem's NDS. That premise is not established and is unlikely to hold: NDS levels are ordinal and depend on the geometry of the A×R metric vectors, which varies with the problem, the number of objectives, and the metric set. For example, a level-1 point on MaF1 (where many algorithms yield non-dominated fronts) may represent a different quality than a level-1 point on MaF3 (where fronts are multimodal). The linear/exponential scores then weight these incomparable counts by level index, so the ranking in Table V is not justified. The single-problem experiment in Section V.B is internally coherent; the flaw is specifically the cross-problem aggregation needed for the paper's central claim of reliability 'regardless of the objectives, metrics, and algorithms employed'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a method for ranking multi-/many-objective optimization algorithms using multiple performance indicators. For each algorithm and run, M indicator values form an M-dimensional point; all A×R points are pooled and subjected to non-dominated sorting (NDS) to assign Pareto levels, and each algorithm's contribution per level is counted. Four ranking techniques (Olympic, linear, exponential, and adaptive) convert these counts into overall ranks. The method is applied to rank ten CEC 2018 algorithms on 15 MaF problems with 5, 10, and 15 objectives using ten indicators, and the resulting ranks are compared with the official CEC rankings based on IGD and HV. The paper claims that the approach yields reliable rankings \"regardless of the objectives, metrics, and algorithms employed\" and that it is parameter-free.","tokens_in":17959,"tokens_out":6074,"duration_ms":51919,"significance":"The idea of treating indicators as objectives and using Pareto sorting to avoid explicit weighting is natural and potentially useful. The construction of NDS levels from per-run metric vectors is clearly described, and the four scoring rules are transparent and reproducible from the reported tables. The paper has the merit of making the full count matrix available, and the sanity check in Section V.E—where the proposed method using only HV and IGD broadly matches the CEC committee ranking—shows that the pipeline is not arbitrary. However, the central reliability claim exceeds what the evidence supports: the only external validation is one competition benchmark, there is no sensitivity analysis, and the cross-problem aggregation at the heart of the overall ranking rests on an unexamined commensurability assumption. The per-problem ranking experiments are sound but do not by themselves justify the \"regardless of objectives/metrics/algorithms\" conclusion.","major_comments":[{"comment":"The overall ranking adds, for each algorithm, the number of NDS points at the same level index l across all 15 MaF problems and across M \\in {5,10,15}. This presupposes that a point on level l in one problem's NDS is equivalent to a point on level l in another problem's NDS. Since NDS levels are ordinal and depend on the shape of the A × R metric-score cloud, which varies with problem geometry, number of objectives, and metric set, the premise is not established. The linear, exponential, and adaptive scores in Eqs. (17), (18), and (21) then weight these aggregated counts by level index, so Table V and the claim in Section I of reliability \"regardless of the objectives, metrics, and algorithms employed\" are not justified. The authors should either provide a commensurability argument (e.g., invariance properties) or restrict the claim to per-problem and per-objective-count rankings.","section":"Section V.D, Tables IV and V"},{"comment":"Two-set coverage C(A,B) is defined in Eq. (7) as a binary indicator comparing two approximation sets, but the proposed method's input matrix in Section IV, Step 4, requires one scalar value per algorithm per run for each of the ten metrics. The paper does not specify how C is turned into a per-run unary score—for example, whether each run is compared against all other runs and averaged, or against a reference set. Without this specification, the ten-dimensional input used in Sections V.B–V.D is not well-defined for one of the ten indicators, and the reported results cannot be reproduced.","section":"Section III and Section IV, Step 4"},{"comment":"The manuscript describes the method as \"parameter-free\" (Sections I and VI), but the four ranking techniques introduce hand-chosen weighting schemes: linear weights L, L−1, ..., 1 in Eq. (17), exponential weights 2^0, 2^{−1}, ... in Eq. (18), and cumulative-ratio weights in Eq. (21). These choices change the final ranking, as the paper itself notes (e.g., AGE-II is 5th by Olympic but 7th by linear in Table V). No sensitivity analysis or selection criterion is provided, so the parameter-free claim is not supported. At minimum, the authors should qualify the claim or show that the rankings are stable under reasonable alternative weight schedules.","section":"Section IV, Eqs. (17)–(21) and Section I"},{"comment":"The reliability claim is not backed by statistical evidence. The results report only point rankings; there are no error bars, significance tests, or sensitivity analyses with respect to the chosen metric set, the number of runs (R=20), the weighting scheme, or the handling of metric scales (e.g., inversion versus sign change). The comparison against CEC2018 in Section V.E is a single case, and the ten-metric versus two-metric rankings differ substantially; the paper attributes this difference to the metric set but does not validate the ten-metric ranking against any ground truth. A sensitivity analysis or a more cautious interpretation is needed before the method can be called reliable \"regardless of\" the inputs.","section":"Section V, Tables III–VI"},{"comment":"The validation in Section V.E compares the proposed method with the CEC ranking only when both use the same two metrics (HV and IGD). That comparison tests the NDS-plus-counting mechanism but not the paper's claimed advantage of using many metrics. The ten-metric ranking is presented without any external validation or comparison against an existing multi-metric ranking procedure, such as the ensemble method of Yen et al. [19]. A meaningful validation would require a comparative baseline or a known ground truth for multi-metric ranking.","section":"Section V.E"}],"minor_comments":[{"comment":"\"ECE2018 committee\" appears to be a typo; it should read \"CEC2018 committee.\"","section":"Section V.E, paragraph 1"},{"comment":"The text states that the decreasing weights are 20, 2^{−1}, 2^{−3}, ..., but the formula uses 2^0, 2^{−1}, 2^{−2}, ...; these are inconsistent and should be aligned.","section":"Section IV, Eq. (18)"},{"comment":"The sentence \"we have a total of nine hundred 10-dimensional points\" is ambiguous: each algorithm contributes 900 points (15 problems × 3 objective counts × 20 runs), so the total across ten algorithms is 9,000 points. Please clarify.","section":"Section V.D, paragraph 1"},{"comment":"Table II lists MaF1 through MaF9 and MaF11 through MaF15 but omits MaF10, although MaF10 is used in Table III and Figure 1; please add the missing row or explain why it is absent.","section":"Table II"},{"comment":"The matrix display after Step 4 contains garbled repeated labels such as \"m1m1m1\" and \"a1,1a1,1a1,1\"; use standard matrix notation for readability.","section":"Section IV, matrix display"},{"comment":"The caption refers to \"10-objective\" and \"15-objective\" MaF problems; for consistency with the rest of the paper, consider writing \"10-objective\" and \"15-objective\" rather than \"10-objective\" and \"15-objective\" if the intended meaning is the number of objectives M.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The per-problem methodology is coherent and the reported counts allow reproducibility of the four scoring rules. The main risk is the cross-problem aggregation in Section V.D, which the paper does not justify; the title and abstract promise a general reliability result that the current evidence cannot support. If the authors narrow the claim to per-problem rankings or provide a rigorous commensurability argument, the paper could become suitable for publication. The lack of any comparison with existing multi-metric ensemble ranking methods is also worth addressing in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about how we rank multi-objective algorithms. The core idea is simple: take per-run vectors of several quality indicators, run non-dominated sorting on them, count each algorithm's points per Pareto level, and aggregate with one of four weighting schemes—Olympic, linear, exponential, and adaptive. That's a clean, parameter-light pipeline, and the adaptive rule is a sensible compromise between looking only at the top level and looking at all levels equally.\n\nWhat is actually new is the particular use of NDS on per-run indicator vectors rather than on a single scalarized index. That is a natural extension of the ensemble idea in [19] and the authors' own prior work, but the paper describes the method clearly and the four counting rules are easy to reimplement.\n\nThe biggest soft spot is exactly what the stress test flags. Section V.D sums level counts across 15 MaF problems and three objective counts to build Table IV, and then Table V's overall ranking uses those sums. That presupposes a 'level 2' point on MaF1 is comparable to a 'level 2' point on MaF13. NDS levels are ordinal and depend on the geometry of the indicator vectors; nothing in the paper justifies this commensurability. So the overall ranking and the 'reliable ranking regardless of objectives, metrics, and algorithms employed' claim are not supported. The single-problem and per-objective-count experiments (V.B and V.C) are internally coherent; the flaw is specifically the cross-problem aggregation.\n\nSecond, validation is thin. No error bars, no significance tests, no sensitivity analysis over metric subsets or level weights, and no comparison with rank-aggregation baselines such as Borda count. The comparison with the CEC 2018 committee is a sanity check, not a validation. The claim that the method is 'parameter-free' is also a bit generous—the choice of weighting scheme is a free parameter.\n\nThird, minor typos: the exponential weights are written as 20, 2−1, 2−3, ...; the second exponent should be 2−2. And the objective counts are given as 3-, 5-, 15 in Section V.A but 5-, 10-, 15 everywhere else.\n\nWho is this for? Researchers who want a simple, explainable way to combine several indicators into one ranking for a competition or leaderboard. The paper deserves a serious referee, but the authors should either drop the strong universal claim or justify the commensurability with an experiment or a proof-of-concept simulation. I'd send it to review; the single-problem results are a reasonable starting point.","headline":"A clean, simple NDS-based multi-metric ranking method that is plausible on single problems, but its cross-problem aggregate ranking rests on an unproven commensurability assumption; worth a serious look with more validation.","tokens_in":18501,"tokens_out":3402,"would_cite":true,"duration_ms":28018,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pareto sorting ranks optimization algorithms without choosing metric weights.","keywords":["multi-objective optimization","many-objective optimization","performance indicators","non-dominated sorting","algorithm ranking","Pareto optimality","multi-metric comparison","CEC 2018 benchmark"],"falsifier":"Build a two-algorithm experiment on a mix of intentionally easy and intentionally hard problems where algorithm A beats B on every easy problem and B beats A on every hard problem. The final ranking sums level counts across problems, so varying only the number of easy versus hard problems in the suite should flip the winning algorithm without any algorithm changing; if it does, the level-count aggregation is not a stable basis for ranking.","tokens_in":17517,"feed_emoji":"🏆","tokens_out":8990,"duration_ms":76093,"temperature":0.7,"pith_summary":"Multi-objective optimizers are usually compared with one quality indicator, but no single indicator captures convergence, diversity, and spread at once and different indicators often disagree. This paper proposes treating each indicator as a separate objective: apply non-dominated sorting to the per-run indicator scores of all algorithms, count how many runs of each algorithm land on each Pareto level, and convert those counts into a ranking with one of four scoring rules. The method needs no combination weights, accepts any number of metrics, and can absorb new indicators as they appear. The paper claims the result is a reliable ranking regardless of the objectives, metrics, and algorithms employed, and demonstrates it on the 2018 CEC competition's ten algorithms by ranking them with ten indicators on 15 MaF problems. When restricted to the two metrics the competition used, the proposed ranking closely tracks the official leaderboard.","feed_headline":"Pareto sorting ranks optimizers with no metric weights","feed_subtitle":"Treating each quality indicator as an objective merges them into a single, weight-free leaderboard.","key_machinery":"The engine is non-dominated sorting (NDS) applied to a matrix of $A \\times R$ points, where $A$ is the number of algorithms, $R$ the number of runs, and each point is an $M$-dimensional vector of indicator scores after sign reversal so all metrics are minimized. NDS strips off successive Pareto fronts, producing level counts $n_{a_i l_j}$ for each algorithm at each level. Four scoring rules convert those counts into ranks: Olympic compares level-1 counts lexicographically level by level; linear weights levels by $L, L-1, \\ldots, 1$; exponential weights them by $2^0, 2^{-1}, \\ldots, 2^{-(L-1)}$; and adaptive sums, over all levels, the ratio of an algorithm's cumulative count at that level to the total cumulative count. The method's parameter-free character comes from relying on dominance structure rather than on user-chosen metric weights.","core_discovery":"The paper's central claim is that the conflict between performance indicators can be turned into a Pareto dominance problem rather than a weighting problem. Each run of an algorithm becomes one point in a space whose coordinates are indicator scores; non-dominated sorting layers these points into levels, and an algorithm is better the more of its runs occupy the early levels. The four scoring methods (Olympic, linear, exponential, and adaptive) are alternative ways to read those counts into a total order. The authors report that with the two official metrics, HV and IGD, their method yields rankings comparable to the 2018 CEC committee's, and with all ten metrics it produces a broader leaderboard headed by fastCAR and closed by RPEA. They conclude that the approach gives a fair, parameter-free multi-metric comparison and can serve any field where entities are judged by several conflicting criteria.","pith_inferences":["Editor's inference: because final ranks sum level counts across problems and objective counts, the method rewards algorithms that score well on easy problems as much as on hard ones; a user comparing across benchmark suites of different difficulty will see rank flips that are not caused by algorithmic change.","Editor's inference: the method counts each run equally, so it does not distinguish a run that just barely reaches level 1 from one that dominates the level; weighting runs by dominance margin or applying $\\varepsilon$-dominance, which the paper mentions as a safeguard, would make the ranking more robust.","Editor's inference: a leave-one-metric-out ablation on the ten indicators would reveal which indicators carry the leaderboard; the paper reports the full ten-metric ranking but not this diagnostic, and it is directly testable from the same per-run scores.","Editor's inference: the same NDS-plus-counting scheme could be used for online algorithm selection or portfolio construction, where many metrics are observed per configuration and one must produce a total order without inventing a scalar objective."],"forward_implications":["A researcher can drop manual weight selection: any set of indicators, old or newly proposed, plugs into the same procedure without changing the method.","Benchmarks become composable: the published per-run indicator values can be re-ranked with different metric subsets to test how sensitive a leaderboard is.","The method transfers to any domain with multiple conflicting quality scores, such as comparing machine learning models on accuracy, latency, and robustness, which the paper lists but does not benchmark.","Because the adaptive rule showed the highest average pairwise correlation with the other rules, the paper recommends it, or the average rank across all four, when a single overall ranking is needed.","If the paper is right, multi- and many-objective algorithm comparisons no longer need to be tied to one champion metric, and the official competition ranking shown in Table VI can be reproduced using only that competition's two metrics."],"supporting_citations":[{"why":"Defines the fast non-dominated sorting procedure that generates the Pareto levels whose counts drive the ranking.","marker":"[23]"},{"why":"Supplies the 15 MaF benchmark problems and the precomputed approximate Pareto fronts of the ten competition algorithms used in the validation.","marker":"[38]"},{"why":"Defines IGD, one of the two official competition metrics used for the direct comparison in Table VI.","marker":"[12]"},{"why":"Defines the hypervolume indicator, the other official metric used for the direct comparison with the CEC 2018 ranking.","marker":"[14]"},{"why":"Provides the rationale that at least as many indicators as objectives are needed to decide whether one solution set is better, motivating the multi-metric design.","marker":"[17]"},{"why":"An earlier ensemble approach that combines multiple metrics into a ranking, serving as the existing baseline the proposed method is positioned against.","marker":"[19]"},{"why":"Defines $\\varepsilon$-dominance, which the paper proposes as a safeguard against algorithms exploiting the NDS dominance relation.","marker":"[40]"}],"fun_headline_variants":["Weight-free Pareto ranking for multi-objective algorithms","Turn every indicator into an objective for fair algorithm ranking","Non-dominated sorting merges metrics into one rank list","Rank optimizers without weighting: Pareto on indicator scores","Four Pareto-scoring methods give a single leaderboard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes a run's Pareto level means the same thing across every test problem and every objective count, so that counts from easy and hard problems can be added up; if that equivalence fails, the totals that settle the final ranking are not measuring one comparable quantity.","fun_headline_variants_meta":{"raw":{"variants":["Weight-free Pareto ranking for multi-objective algorithms","Turn every indicator into an objective for fair algorithm ranking","Non-dominated sorting merges metrics into one rank list","Rank optimizers without weighting: Pareto on indicator scores","Four Pareto-scoring methods give a single leaderboard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000413,"raw_usage":{"total_tokens":2164,"prompt_tokens":1005,"completion_tokens":1159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1084}},"tokens_in":621,"tokens_out":1159,"duration_ms":10332,"temperature":1.0,"reasoning_tokens":1084,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:37:20.700274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a two-algorithm experiment on a mix of intentionally easy and intentionally hard problems where algorithm A beats B on every easy problem and B beats A on every hard problem. The final ranking sums level counts across problems, so varying only the number of easy versus hard problems in the suite should flip the winning algorithm without any algorithm changing; if it does, the level-count aggregation is not a stable basis for ranking.","supporting_citations":[{"cited_title":"A fast and elitist multiobjective genetic algorithm: NSGA-II,","cited_arxiv_id":null,"evidence_quote":"Defines the fast non-dominated sorting procedure that generates the Pareto levels whose counts drive the ranking."},{"cited_title":"A benchmark test suite for evolutionary many-objective optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the 15 MaF benchmark problems and the precomputed approximate Pareto fronts of the ten competition algorithms used in the validation."},{"cited_title":"Solving multiobjective optimization problems using an artificial immune system,","cited_arxiv_id":null,"evidence_quote":"Defines IGD, one of the two official competition metrics used for the direct comparison in Table VI."},{"cited_title":"Multiobjective evolutionary algorithms: a com- parative case study and the strength pareto approach,","cited_arxiv_id":null,"evidence_quote":"Defines the hypervolume indicator, the other official metric used for the direct comparison with the CEC 2018 ranking."},{"cited_title":"Performance assessment of multiobjective optimizers: An analysis and review,","cited_arxiv_id":null,"evidence_quote":"Provides the rationale that at least as many indicators as objectives are needed to decide whether one solution set is better, motivating the multi-metric design."},{"cited_title":"Performance metric ensemble for multiobjective evolutionary algorithms,","cited_arxiv_id":null,"evidence_quote":"An earlier ensemble approach that combines multiple metrics into a ranking, serving as the existing baseline the proposed method is positioned against."},{"cited_title":"A new evolutionary decision theory for many-objective optimization problems,","cited_arxiv_id":null,"evidence_quote":"Defines $\\varepsilon$-dominance, which the paper proposes as a safeguard against algorithms exploiting the NDS dominance relation."}],"review_version":1}