{"id":"573d3e26-ef7d-4efe-92f5-5a7cdb315ca0","arxiv_id":"2504.12803","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using SHAP-based explanations from an adapted IOHxplainer, the paper finds that lower cognitive coefficients help PSO and that Ring topology shows the best in-sample R2, but only on 2D benchmarks with no validation.","lead":"This paper applies an explainable AI tool, IOHxplainer, to study how three Particle Swarm Optimization communication topologies, Ring, Star, and Von Neumann, change search behavior on benchmark functions. A generalist might read it to see whether explainability tools can produce practical guidance for choosing optimizer settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ring-topology and c1 guidelines rest on an undefined in-sample R2 and qualitative SHAP readings, with no significance tests or c1-level performance tables; the evidence does not yet support the central practical claims.","rationale":"The reader's weakest assumption is that the 2D, 100-iteration experiments generalize to the optimization tasks targeted by the guidelines. I agree that this is a real limitation, and the paper itself lists higher-dimensional scaling as future work. However, the more immediate problem is that the evidence is not yet convincing even for the tested setting: Table 2's topology ranking relies on an undefined 'R2 train' column, no statistical tests or confidence intervals are reported, and the c1 guideline is presented as a qualitative SHAP reading rather than as a direct comparison of AOCC across c1 levels. To the paper's credit, the GitHub repository and the described BBOB/instance/replicate protocol mean the missing analyses are checkable rather than irreproducible. If the concrete test shows significant Ring advantages and a monotone c1 effect, the current CONDITIONAL verdict can stand; if not, the central practical claims would need to be substantially weakened. I therefore keep the reader's CONDITIONAL verdict, while noting that the justification should shift from 'generalization is unproven' to 'the reported evidence does not yet establish the claims even in 2D'.","tokens_in":7096,"tokens_out":8611,"duration_ms":97022,"concrete_test":"Recompute the Section 4.1 topology comparison from the archived GitHub data using actual AOCC performance, not R2 train: for each of the 24 BBOB functions, take the best configuration per topology, and run a paired Wilcoxon signed-rank test (or report 95% bootstrap confidence intervals on the difference) for Ring versus Star and Ring versus Von Neumann. Also, for Section 4, compute the mean AOCC for each c1 level (0.3, 0.5, 0.7, 0.9) after averaging over all other configuration parameters, per function and per topology. If the Ring advantage is not significant at alpha = 0.05, or if c1 = 0.3 is not best in a clear majority of function/topology cells, the corresponding guideline fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the paper's two headline guidelines are not backed by a defined, quantitative comparison. In Section 4.1, the conclusion that the Ring topology is the superior choice for regression tasks is based on the Table 2 column 'R2 train', but R2 train is never defined and appears to be an in-sample goodness-of-fit statistic rather than an optimizer-performance metric; the actual performance columns (single-best mean, avg-best mean, all mean) differ between topologies by only about 0.001 to 0.03 and have overlapping standard deviations. No significance test, confidence interval, or held-out evaluation is reported, so the ranking could be noise. Likewise, Section 4's statement that 'a lower c1 value consistently improves performance, while a larger c1 worsens it' is drawn from SHAP swarm plots without reporting mean AOCC per c1 level, per function, or per topology; 'consistently' cannot be verified from the text. These issues are more basic than the 2D/100-iteration transfer concern because they affect the validity of the conclusions even within the tested setting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies an adapted IOHxplainer framework with SHAP to study how three PSO communication topologies (Ring, Star, Von Neumann) affect performance on the 24 noiseless BBOB functions at dimension 2 with a budget of 100 iterations. It reports AOCC-based performance for 1,728 configurations, presents SHAP swarm plots to interpret hyperparameter effects, and derives practical guidelines: lower c1 improves performance, and Ring topology is best for regression tasks based on an R2 value. The authors also report runtime comparisons among topologies and provide a GitHub repository with data.","tokens_in":7248,"tokens_out":1847,"duration_ms":20966,"significance":"If the central claims were established, the paper would provide useful, practically actionable guidance for topology and hyperparameter selection in PSO, and it would demonstrate a promising use of XAI tools for algorithm configuration. Strengths include the use of a standard BBOB benchmark setup, a clearly described experimental grid, and the availability of code and full results in a public repository. The main limitation is that the headline conclusions currently rest on undefined or in-sample performance summaries and on qualitative SHAP readings, with no significance testing; the conclusions therefore need substantially stronger quantitative support before they can be relied upon.","major_comments":[{"comment":"The conclusion that 'the Ring topology is the superior choice for regression tasks focused on maximizing the R2 value' is based on the column labeled 'R2 train', but this metric is never defined. As labeled, it appears to be an in-sample goodness-of-fit statistic, not an optimizer performance measure, so its relationship to AOCC and to regression-task suitability is unclear. Moreover, the AOCC differences between topologies in the 'all mean' column are typically 0.001 to 0.03 with overlapping standard deviations, so without a definition of the regression target, a significance test, or a held-out evaluation, the Ring-topology ranking cannot be considered established.","section":"Section 4.1, Table 2"},{"comment":"The statement that 'a lower c1 value consistently improves performance, while a larger c1 worsens it' is supported only by qualitative SHAP swarm plots. The text does not report mean AOCC per c1 level, per function, per topology, or any statistical comparison across c1 values, so 'consistently' is not verifiable from the reported evidence. The SHAP attributions themselves are not validated: the paper does not specify the model being explained, the background distribution, the baseline value, or any measure of attribution accuracy, so the SHAP plots cannot substitute for a direct quantitative analysis.","section":"Section 4, hyperparameter claims"},{"comment":"The 'single-best' and 'avg-best' configurations are selected on the same benchmark runs whose AOCC values are then reported in Table 2. This in-sample selection makes the reported means optimistic and the comparisons between topologies potentially circular. The paper should either perform a separate evaluation of the selected configurations on independent runs or report paired differences with confidence intervals across the 24 functions, as the current presentation does not allow the reader to distinguish genuine topology effects from selection noise.","section":"Section 4.1, single-best and avg-best selection"},{"comment":"All experiments are run only in 2 dimensions with a budget of 100 iterations, and Section 5 explicitly lists 'scaling to higher dimensions' as future work, yet the conclusions are phrased as general guidelines for choosing topologies and hyperparameters for different types of optimization problems. The paper provides no evidence that the c1 effect or the topology ranking persists at higher dimensions or larger budgets, so the practical guidelines should either be restricted to the tested regime or accompanied by at least one higher-dimensional check.","section":"Section 3 and Section 5"}],"minor_comments":[{"comment":"The reference to 'AOCC 3' in the text should be 'AOCC (Equation 3)' or a citation to the metric definition, rather than an unexpanded label.","section":"Section 2.2, Figure 1"},{"comment":"The 'all mean' value of 1.36E-01 for Von Neumann on f12 appears to be an outlier that is inconsistent with the other values in that row; please verify the decimal point and the underlying data.","section":"Table 2, f12 Von Neumann row"},{"comment":"The paper would benefit from citing the original IOHxplainer paper and the IOHexperimenter/COCO documentation more precisely, as the current references do not clearly identify where the adapted framework originates.","section":"Section 1, references"},{"comment":"The text contains several colloquial or vague phrases, such as 'promoting structure' for the Von Neumann topology and 'you can find a more detailed explanation of the framework in Figure 1'; these should be tightened to formal, precise descriptions.","section":"Throughout"},{"comment":"The runtime comparison is based on wall-clock hours on a single machine, which is sensitive to machine load and implementation details; consider reporting per-run CPU time or normalized runtime to make the comparison reproducible.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a potentially useful application of XAI to PSO configuration, but the central practical claims are currently supported by in-sample metrics and qualitative plots rather than by rigorous quantitative evidence. The issues are fixable within the scope of the manuscript: define R2, add held-out or paired statistical evaluation, validate the SHAP analysis, and either add a higher-dimensional experiment or substantially temper the generalization claims. I therefore see this as a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a 4-page GECCO Companion paper that applies IOHxplainer/SHAP to PSO hyperparameters under three topologies. It's a legitimate new application, and the authors provide code and data. But the two practical guidelines — \"lower c1 always helps\" and \"Ring is best for regression tasks\" — are not actually supported by the numbers as reported. The stress-test note is on target: R2 train appears in Table 2 but is never defined, and the c1 claim is made from SHAP swarm plots without any per-c1 performance table.\n\nWhat's new: using SHAP attribution on PSO topology/hyperparameter search is a reasonable and fairly novel use of an existing tool. The experimental setup is standard (BBOB, 24 functions, 5 instances, 5 seeds) and reimplementation-friendly. The paper is honest about its limits: it explicitly says scaling to higher dimensions is future work, and the GitHub link is a plus.\n\nWhat's soft: (1) The central topology ranking relies on R2 train, which is undefined; if it's an in-sample regression R2 from the hyperparameter model, then it's not evidence of optimization performance. The actual AOCC means differ by ~0.001-0.03 with overlapping stds, so the \"Ring is superior\" conclusion could be noise. (2) No significance tests or confidence intervals anywhere, despite \"statistical analysis\" in the abstract. (3) Single-best and avg-best configs are selected and evaluated on the same runs, so the comparisons are in-sample. (4) The c1 claim isn't backed by a table of mean AOCC per c1 level per function; \"consistently\" is doing too much work. (5) The 2D/100-iteration limit is acknowledged, but that doesn't fix the within-setting issues.\n\nThere's also an apparent typo in Table 2 f12 Von Neumann \"all mean\" (1.36E-01 vs 1.35E-02), which makes me trust the table a bit less.\n\nWho it's for: practitioners who want a quick, qualitative read on how PSO hyperparameters interact with topology, and people working on XAI-for-algorithms. It's not yet reliable as quantitative guidance.\n\nRecommendation: send to peer review as a borderline. The flaws are fixable with held-out validation, per-level tables, significance tests, and defining R2. For a 4-page companion, the contribution is modest but real.","headline":"A reproducible but thinly supported XAI-for-PSO companion: the new application is fine, but the headline guidelines rest on an undefined R2 metric and in-sample comparisons.","tokens_in":7841,"tokens_out":2432,"would_cite":false,"duration_ms":24630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a particle swarm’s communication topology changes which hyperparameter settings work best, and derives task-specific rules for choosing topology and coefficients.","keywords":["particle swarm optimization","communication topology","ring topology","star topology","grid topology","explainable AI","hyperparameter analysis","benchmark functions"],"falsifier":"Run the same 1,728-configuration sweep in dimensions 5, 10, and 30 with a 1,000-iteration budget and check whether low $c_1$ still dominates and whether the ring topology still gives the highest $R^2$; either flip would refute the paper’s guidance.","tokens_in":6849,"feed_emoji":"🐝","tokens_out":8435,"duration_ms":83481,"temperature":0.7,"pith_summary":"This paper tries to show that a particle swarm optimizer’s communication topology is not a neutral detail: it changes which hyperparameters matter, how fast the swarm converges, and which problem shapes the algorithm handles well. The authors run an explainable benchmarking pipeline over 1,728 configurations of a 2-dimensional, 24-function benchmark suite and use attribution values to read off the effect of each setting. Their reported pattern is consistent: a low cognitive coefficient $c_1$ helps and a high one hurts, small swarms suit unimodal problems under the ring topology while larger swarms suit multimodal ones, and the ring topology gives the best fit for regression-style tasks. If these findings hold, practitioners could replace trial-and-error tuning with task-class rules for choosing topology and coefficients.","feed_headline":"Ring topology tops regression PSO; low c1 always helps","feed_subtitle":"An explainable sweep of 1,728 configurations maps hyperparameters to task types, replacing trial-and-error tuning.","key_machinery":"The central object is the communication topology—the graph that decides which particles exchange their best-known positions—because it determines which term in the PSO velocity update dominates the search. The argument runs through an adapted explainable benchmarking pipeline: configurations are sampled from the hyperparameter grid, each is run on the 24-function benchmark suite, performance is scored by the area over the convergence curve (AOCC), and attribution values are computed to isolate each hyperparameter’s marginal contribution. That pipeline is what converts raw convergence numbers into the paper’s configuration rules.","core_discovery":"On its own terms, the paper establishes that the effect of each PSO hyperparameter depends on communication topology. In the velocity update $v_i(t+1)=w v_i(t)+c_1 r_1(p_{best,i}-x_i(t))+c_2 r_2(g_{best}-x_i(t))$, the $c_1$ term is the consistent driver: lower values of $c_1$ improve area-over-the-convergence-curve performance across functions and topologies, while larger values worsen it. The paper reports that the ring topology yields the highest $R^2$ when fitting performance across the benchmark set, making it the recommended choice for regression-oriented tasks, while the star topology converges fastest but risks early stagnation and the grid topology is the most time-efficient. For unimodal functions a small swarm of 50 particles works best in the ring, for multimodal functions a larger swarm of 100 works better, and the star benefits from small swarms on highly multimodal functions.","pith_inferences":["Because the attribution signs flip across topologies, a single global hyperparameter default is likely suboptimal; an automated tuner that conditions on topology should outperform one that ignores it.","The $R^2$ used to rank topologies is fit on the benchmark evaluation itself, so the ring recommendation should be tested on held-out functions or real regression problems before being adopted as a general rule.","The same pipeline could test adaptive or dynamic topologies, for example starting with the star for fast progress and switching to the ring later, to see whether the reported trade-offs can be combined in one run."],"forward_implications":["For unimodal problems, use a small swarm ($n=50$) with the ring topology; for multimodal problems, use a larger swarm ($n=100$), and keep $c_1$ low.","The ring topology is the best default among the three for regression-style tasks where fit quality, measured by $R^2$, is the goal.","The star topology should be expected to converge quickly but to stagnate on difficult multimodal landscapes, making it a poor choice when local optima are dense.","The grid topology is the most compute-efficient information-sharing structure at this budget, useful when per-run cost matters.","Attribution-based explanation turns PSO configuration into readable rules: each hyperparameter’s effect can be stated as a sign and magnitude rather than a tuned value."],"supporting_citations":[{"why":"Defines the particle swarm velocity and position update rules that every configuration in the study instantiates.","marker":"[7]"},{"why":"Supplies the 24 noiseless benchmark functions whose performance scores drive all comparisons.","marker":"[5]"},{"why":"Provides the attribution method used to isolate each hyperparameter’s marginal contribution to performance.","marker":"[10]"},{"why":"Defines the anytime-performance evaluation that underlies the area-over-the-convergence-curve metric used as the outcome.","marker":"[4]"},{"why":"Motivates the claim that topology choice directly shapes exploration and exploitation in particle swarm optimization.","marker":"[9]"},{"why":"Classifies population topologies, including the structured grid topology used in the comparison.","marker":"[11]"},{"why":"Supports the expected behavior of the ring topology on multimodal problems.","marker":"[14]"}],"fun_headline_variants":["Low c1 always helps; ring tops regression","Ring for regression, low c1 for robust PSO","Explainable PSO: ring fits regression, low c1 wins","Topology verdict: ring for regression, low c1 universally"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guidelines rest on the assumption that swarm behavior in two dimensions with a 100-iteration budget transfers to the higher-dimensional, longer-running optimization tasks that practitioners actually face.","fun_headline_variants_meta":{"raw":{"variants":["Low c1 always helps; ring tops regression","Ring for regression, low c1 for robust PSO","Explainable PSO: ring fits regression, low c1 wins","Topology verdict: ring for regression, low c1 universally"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1238,"prompt_tokens":871,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":299}},"tokens_in":487,"tokens_out":367,"duration_ms":4413,"temperature":1.0,"reasoning_tokens":299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:22:03.459833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,728-configuration sweep in dimensions 5, 10, and 30 with a 1,000-iteration budget and check whether low $c_1$ still dominates and whether the ring topology still gives the highest $R^2$; either flip would refute the paper’s guidance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classifies population topologies, including the structured grid topology used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the expected behavior of the ring topology on multimodal problems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the particle swarm velocity and position update rules that every configuration in the study instantiates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 24 noiseless benchmark functions whose performance scores drive all comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the anytime-performance evaluation that underlies the area-over-the-convergence-curve metric used as the outcome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the claim that topology choice directly shapes exploration and exploitation in particle swarm optimization."}],"review_version":1}