{"id":"acff8c3e-64a0-4b37-b741-27efa1bf3eea","arxiv_id":"2512.18343","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 65-algorithm benchmark on a bi-objective repairable redundancy-allocation problem shows that algorithm rankings are budget-dependent and that Scaled Binomial Initialization changes relative performance.","lead":"This paper benchmarks 65 multi-objective optimization algorithms on a redundancy-allocation problem where the goal is to balance cost and availability in repairable systems. It finds that the best algorithm depends strongly on how many evaluations you can afford, and that a specialized initialization gives many methods a large head start.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's warm-standby failure rate uses active count instead of standby count; if implemented, every availability value and the resulting rankings are wrong.","rationale":"The reader's weakest assumption was the correctness of the CTMC transition models and system-level availability formulas. I identified a concrete, textually evidenced error in Algorithm 1: the warm-standby failure rate is written as the product of active component count and standby failure rate, not standby count. This is exactly the class of error that would propagate invisibly into every performance indicator and every algorithmic ranking, because the entire benchmark is built on these availability values. It is not an external-consensus disagreement; it is an internal inconsistency between the pseudocode and the model description. The central claim about budget-dependent rankings and strategy preferences cannot survive if the objective function is wrong. The test is straightforward because the code is publicly available: inspect the relevant line and, if necessary, recompute a small case. I therefore keep the reader's CONDITIONAL verdict: the paper should not be rejected solely on this suspicion, but it should not be accepted as definitive until the code is checked and the pseudocode corrected or shown to be a harmless typo.","tokens_in":36353,"tokens_out":8791,"duration_ms":89376,"concrete_test":"Download the Zenodo repository (doi:10.5281/zenodo.17981720), locate the CTMC construction for warm and mixed strategies, and check whether the warm-standby failure transition rate is computed as standby_count * lambda_standby or active_count * lambda_standby. Independently re-derive the transition matrix for a single 2-out-of-3 warm-standby subsystem with both variants and compare the steady-state availability. If the active-count version is used, recompute HV for at least one CS/W combination with corrected rates and check whether the ranking (e.g., NSGA-II+ARSBX-SBI vs CMOPSO-SBI) flips. If the code already uses standby count, this is a pseudocode typo and no ranking change is expected.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claims—budget-dependent rankings, SBI advantage, NSGA-II+ARSBX-SBI robustness, and hot/mixed dominance—are all derived from hypervolume computed over subsystem availabilities produced by the CTMC construction in §3.2/Algorithm 1. In that pseudocode, the transition for a warm-standby failure (lines 41–46) is written as ΛS(i,j) ← S(i,1)·λstandby, i.e., the number of active components times the standby failure rate. The state vector is explicitly defined as (active, standby, failed), so S(i,1) is active count and S(i,2) is standby count; §3.2.3 states that failure rates are sums over 'all active or standby units'. The correct rate is S(i,2)·λstandby. The same logic is used correctly for active failures in line 40 (S(i,1)·λactive). If the released MATLAB code implements the pseudocode literally, every warm and mixed subsystem availability is mis-specified. Since Table 4 shows mixed and hot strategies dominate the Pareto fronts and warm/mixed components appear throughout the high-availability solutions, the error would propagate into every HV value, every algorithm ranking, and the paper's headline conclusions. At minimum, the manuscript is internally inconsistent: from the text alone, the objective functions are not verified correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale benchmark of 65 multi-objective metaheuristics on a bi-objective redundancy allocation problem for repairable k-out-of-n systems. The problem minimizes system cost and maximizes steady-state availability, with subsystem decisions covering the number of redundant components and the choice among cold, warm, hot, or mixed standby strategies. Subsystem availability is computed via continuous-time Markov chains (CTMCs). A controlled experimental campaign uses six case studies of varied structure and dimension, four weight limits per case, two initialization regimes (random and the authors' SBI), and a budget of 2e6 function evaluations with 10 runs per configuration. Performance is compared via hypervolume, budget-dependent convergence profiles, and non-parametric statistics (Friedman ranks, Wilcoxon signed-rank tests, Holm-Bonferroni correction). The central claims are that algorithm rankings are strongly budget-dependent, SBI provides a substantial early advantage and changes rankings, NSGA-II+ARSBX-SBI is robust for medium and large budgets, and hot/mixed redundancy strategies dominate Pareto-optimal solutions.","tokens_in":36680,"tokens_out":4053,"duration_ms":43726,"significance":"If the underlying CTMC and system-availability evaluations are correct, this is a valuable and unusually comprehensive benchmark: 65 algorithms, 24 problems, reproducible code and aggregate results on Zenodo, and a statistically disciplined comparison protocol. The budget-dependent ranking result and the SBI initialization effect have direct practical implications for algorithm selection in RAP studies. The paper also contributes a new real-world-inspired test suite for multi-objective optimization. However, the significance is conditional on the correctness of the objective-function implementation, which is precisely the point of concern raised in the major comments.","major_comments":[{"comment":"The pseudocode for a warm-standby failure sets the transition rate to ΛS(i,j) ← S(i,1)·λstandby. Since the state vector is defined as (active, standby, failed), S(i,1) is the active count and S(i,2) is the standby count. The text in §3.2.3 states that failure rates are sums over 'all active or standby units', so the standby failure rate should be S(i,2)·λstandby, not S(i,1)·λstandby. This is not a cosmetic issue: Table 4 shows warm and mixed strategies dominate Pareto fronts, so every hypervolume value and ranking in Figs. 12–17 depends on these rates. Please verify the MATLAB implementation, correct the pseudocode and code, and re-run the experiments if the code literally follows the current pseudocode.","section":"§3.2.5, Algorithm 1, lines 41–46"},{"comment":"The manuscript provides no independent validation of the CTMC-based availability values or the system-level availability formulas for CS2, CS3, CS5, and CS6. Given that the entire benchmark ranks algorithms by hypervolume computed from these objective values, the authors should verify their implementation against an independent solver (e.g., numerical solution of the global balance equations or Monte Carlo simulation) for at least several subsystem configurations and all system formulas. The internal inconsistency in Algorithm 1 (above) makes this verification especially important, as it indicates that the implementation may deviate from the intended model.","section":"Eqs. (8)–(13) and §3.2"}],"minor_comments":[{"comment":"The last argument of f should be Am, not A1: it currently reads 'A1(nm, km, rm)', which is a typo.","section":"Eq. (1)"},{"comment":"The 'Acknowledgements' heading appears twice. One duplicate should be removed.","section":"After the References"},{"comment":"The table is visually fragmented across CS blocks with different W values. Please unify the format or add clear subheadings so the reader can identify the columns for each case study and weight limit without scanning back and forth.","section":"Table 4"},{"comment":"The relative-distance analysis uses the best-found HV over all methods/runs as a proxy for the true Pareto front. This is acknowledged, but the paper should also report variance or confidence intervals across the 10 runs, since the 'virtual best' curve is derived from the same data and may overstate the stability of the rankings.","section":"§4.3, Fig. 14"}],"recommendation":"major_revision","confidential_remarks":"The warm-standby CTMC issue in Algorithm 1 is potentially fatal if the released MATLAB code implements the pseudocode literally. This must be resolved before the paper can be accepted. The rest of the benchmarking design is sound and the contribution is useful, but the correctness of the objective function is the foundation of every claim in the paper. Please ask the authors to provide the corrected code together with a validation result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before trusting its headline claims: it's a serious, large-scale benchmark of 65 multi-objective metaheuristics on a repairable k-out-of-n redundancy allocation problem, with six case studies, four redundancy strategies, and a careful budget-dependent analysis. The statistical protocol (Friedman ranks, Wilcoxon, Holm-Bonferroni) is appropriate, and the authors release code and aggregate results. The finding that algorithm rankings change with evaluation budget, and that SBI initialization gives a real head start, is new and worth taking seriously. The paper is not a hype job; it's a solid piece of empirical work.\n\nBut there's a soft spot that could be load-bearing. In Algorithm 1, the warm-standby failure transition (line 45) is written as S(i,1)*(lambda_standby), using the number of active components. The state vector is explicitly (active, standby, failed), and the condition for that branch is that standby count decreases while failed count increases. The rate should be S(i,2)*(lambda_standby) — the number of standby components. The text in Section 3.2.3 says the failure rate is the sum over all active or standby units, which is the correct rule. So the pseudocode is internally inconsistent with the text. If the released MATLAB code implements the pseudocode literally, every warm and mixed subsystem availability is wrong, and every hypervolume-based ranking and the strategy-preference conclusions (hot and mixed dominating) are contaminated. If the code does it correctly, then the pseudocode is just a typo — but the reader can't tell from the manuscript alone.\n\nThe other concerns are minor by comparison: the paper calls best-found solutions 'Pareto-optimal' without a true front, only 10 runs per configuration, and only aggregate data are public (500GB raw data not uploaded). Those are standard limitations, not flaws. The CTMC formulas for the system structures (Eqs. 8–13) are not independently verified, but they're the kind of thing a referee can check.\n\nSo: who is this for? Anyone benchmarking algorithms on real-world-inspired problems, and RAP practitioners who want guidance on algorithm selection. It deserves a serious referee, but the referee must check the actual code against Algorithm 1 before any recommendation can be accepted. My advice: send it to review, and make the code inspection a mandatory part of the process.","headline":"A genuinely useful large-scale benchmark, but a possible load-bearing typo in the warm-standby CTMC pseudocode needs checking before the rankings can be trusted.","tokens_in":37143,"tokens_out":1884,"would_cite":false,"duration_ms":20528,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90B25","90C29","90C59"],"pacs":[],"model":"deepseek-v4-flash","headline":"A single best algorithm cannot be named for redundancy allocation; the winner changes with the evaluation budget and the initialization method.","keywords":["redundancy allocation problem","k-out-of-n systems","multi-objective optimization","metaheuristics","continuous-time Markov chain","standby redundancy","hypervolume","budget-aware benchmarking"],"falsifier":"Run discrete-event simulation on a representative set of subsystem configurations covering all four standby strategies and various n and k values under the paper's failure, repair, and switch rates, then compare steady-state availability to the CTMC predictions; any systematic mismatch would undermine the objective values and therefore every ranking built on them.","tokens_in":36247,"feed_emoji":"⚙️","tokens_out":3839,"duration_ms":40573,"temperature":0.7,"pith_summary":"This paper tries to show that choosing an optimizer for the redundancy allocation problem cannot be reduced to a single winner. Running 65 metaheuristics on six repairable k-out-of-n system benchmarks under two initialization regimes and multiple weight limits, it finds that the ranking of algorithms changes strongly with the number of function evaluations allowed. A structured initialization (SBI) gives a large early advantage and can flip the ordering of methods. It also finds that hot standby and a mixed active-warm standby strategy dominate the cost-availability Pareto fronts, while cold and warm standby essentially never appear. If true, this gives engineers a practical rule: pick the optimizer by budget regime and by whether structured initialization is available.","feed_headline":"Best optimizer for redundancy allocation depends on budget","feed_subtitle":"A 65-method benchmark shows evaluation budget and initialization change algorithm rankings, so choose accordingly.","key_machinery":"The central mechanism is Scaled Binomial Initialization (SBI), a structured way to generate the initial binary population by varying the probability of a 1-bit with the index of the individual, plus continuous-time Markov chain (CTMC) availability models for each standby strategy. The CTMC models convert each subsystem's (n, k, strategy) choice into steady-state availability; hypervolume computed from the resulting bi-objective front feeds the budget-dependent rankings. SBI is what gives methods an early advantage and changes their relative performance, while the CTMC models provide the objective values on which all rankings rest.","core_discovery":"On its own terms, the paper establishes that for the bi-objective redundancy allocation problem in repairable k-out-of-n systems, algorithm performance is a function of the computational budget. Using 65 metaheuristics, six system configurations, four weight limits, and up to 2e6 evaluations per run, it finds hypervolume rankings that shift across budget regimes. SBI initialization provides a substantial head start—in some cases the SBI initial population alone sits near the best-found front—and it changes the ranking of methods, sometimes drastically. The paper's concrete best performers are NNIA-SBI and CMOPSO-SBI for tight budgets, CMODEFTR-SBI and DSPCMDE-SBI for medium budgets, and NSGA","pith_inferences":["Editorial inference: if the budget-dependence pattern holds beyond RAP, then any multi-objective benchmark that reports a single final-budget ranking is likely hiding which algorithms are actually useful for a practitioner's real, fixed evaluation budget.","Editorial inference: SBI's benefit is tied to binary encodings; the same idea—a parameterized, structured initial population—might transfer to other discrete or mixed-integer problems, but that transfer is not tested in this paper.","Editorial inference: the near-absence of cold and warm standby in Pareto fronts is contingent on the paper's parameter values; if cold-switch times were much faster or warm-standby failure rates much lower, those strategies could become competitive.","Editorial inference: because SBI changes rankings so drastically, cross-study comparisons of algorithms that use different initialization schemes may be implicitly unfair; a standardized initialization protocol may be needed before broad algorithmic conclusions can be drawn."],"forward_implications":["Algorithm rankings for RAP should be reported with the evaluation budget explicitly conditioned; a single final-budget ranking can mislead practitioners.","SBI should be used in binary-encoded RAP settings because it consistently raises hypervolume and can put a method in a near-final convergence state from the start.","Hot standby is preferred under tight weight limits, while the mixed active-warm strategy dominates when more spares can be allocated, making cold and warm standby poor design choices under the tested parameters.","Larger systems require roughly an order of magnitude more evaluations to approach the best-found Pareto fronts, so budget planning should scale with system size.","The six case studies form a difficulty gradient that could serve as a reusable real-world benchmark suite for multi-objective optimizers."],"fun_headline_variants":["Best optimizer shifts with budget in 65-method benchmark","Budget flips algorithm rankings in redundancy allocation","Initialization can outrank algorithm choice in RAP","Tight-budget winners differ from long-budget winners","65-method RAP benchmark: budget decides champion"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the CTMC transition models for cold, warm, mixed, and hot standby—together with the system-level availability formulas used in the six case studies—produce correct availability values, because every algorithm ranking is computed from hypervolume of solutions evaluated with those values.","fun_headline_variants_meta":{"raw":{"variants":["Best optimizer shifts with budget in 65-method benchmark","Budget flips algorithm rankings in redundancy allocation","Initialization can outrank algorithm choice in RAP","Tight-budget winners differ from long-budget winners","65-method RAP benchmark: budget decides champion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3079,"prompt_tokens":847,"completion_tokens":2232,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2170}},"tokens_in":591,"tokens_out":2232,"duration_ms":16075,"temperature":1.0,"reasoning_tokens":2170,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:59:24.847189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run discrete-event simulation on a representative set of subsystem configurations covering all four standby strategies and various n and k values under the paper's failure, repair, and switch rates, then compare steady-state availability to the CTMC predictions; any systematic mismatch would undermine the objective values and therefore every ranking built on them.","supporting_citations":[],"review_version":1}