REVIEW 2 major objections 4 minor 12 references
Comparing Optimization Algorithms Through the Lens of Search Behavior Analysis
T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that applying the crossmatch statistical test to the populations of solutions explored by optimization algorithms at each iteration yields a quantitative similarity measure that can distinguish 114 metaheuristics by…
desk verdict A fresh and useful idea for comparing search behavior, but the central statistical claim rests on a violated independence assumption, so the p-values and clustering are not trustworthy as presented. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the crossmatch statistic: given two samples of points, combine them, pair all points to minimize total within-pair distance, and count how many pairs contain one point from each sample; under the null that the samples come from the same distribution, this count has an exact, distribution-free null distribution. The paper uses the test's p-value at each iteration as a local verdict on whether two algorithms are exploring the same region, then averages the failure-to-reject rate over problems and runs to obtain a pairwise similarity value. A secondary mechanism is agglomerative hierarchical clustering on those similarity values, which produces the algorithm groupings shown in the dendrogram.
What would settle it
Run the same pairwise comparisons with each algorithm assigned an independently generated initial population, for example by restarting each algorithm with a different random seed before recording trajectories, and watch whether the similarity dendrogram changes. If the clusters and rejection rates shift substantially, the shared initial population is a material source of dependence; if they stay stable, the crossmatch similarity is robust to that design choice. A more direct check is to take one reported pair, such as BaseDE and SADE on a single BBOB instance, and compare the observed crossmatch statistic at a fixed iteration against the distribution of the statistic obtained by randomly permuting labels across the two shared-seed populations; a large discrepancy would show the independence assumption is violated.
Extended reading notes
Core claim
The central discovery, stated on the paper's own terms, is that the crossmatch test can function as a search-behavior fingerprint for optimization algorithms. The test pools the two populations of candidate solutions from a fixed problem instance, iteration, and run, pairs all points to minimize total within-pair distance, and counts how many pairs contain one point from each algorithm; a small count of crossmatches relative to its exact null distribution rejects the idea that both algorithms are sampling the same distribution. The paper aggregates, over all problems and runs, the fraction of iterations for which the null hypothesis is not rejected, and presents this as a pairwise similarity score. In the resulting dendrogram, variants of the same base algorithm cluster together (differential evolution variants, genetic algorithm variants, and several equilibrium-optimizer and ecosystem-optimizer families), while some algorithms from different MEALPY groups also merge, which the paper takes as evidence that metaphor family does not determine search behavior. The intended use is to let researchers position a new metaheuristic against this portfolio by comparing trajectories rather than descriptions.
Load-bearing premise
The comparison assumes that the populations of two algorithms at the same iteration and run are independent samples, even though all algorithms start from the same initial population under the same random seed; if that dependence matters, the p-values and the similarity clusters built from them are not statistically valid.
Editorial extensions
If this is right
- A new metaheuristic can be checked against the existing 114-algorithm portfolio by comparing its trajectory populations: a high similarity score to an established method would signal that the claimed novelty needs a closer look.
- Researchers can use the crossmatch statistic and the rejection-rate score as complementary indicators: one shows how many points are cross-matched, the other how often the null hypothesis survives.
- Algorithms from different metaphor families that cluster together become candidates for shared underlying mechanisms, worth investigating beyond their labels.
- Families with many variants, such as differential evolution and genetic algorithms, are empirically confirmed to occupy the same behavioral region on BBOB in dimensions 2 and 5.
Reading between the lines
- A natural extension is to test whether the similarity clusters predict performance on unseen problems; if two algorithms cluster together but diverge in final solution quality, behavioral similarity and optimization performance are measuring different things.
- The shared-seed design means the reported p-values may be systematically distorted by dependence; re-running with independent initial seeds would settle whether the dendrogram is an artifact of that design.
- With population size 50 and dimensions 2 and 5, Euclidean pairing is relatively informative; at higher dimensions the nearest-neighbor structure changes, so the same heuristic may need a different distance or dimension reduction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new empirical technique for comparing black-box optimization algorithms by applying the crossmatch multivariate two-sample test to the candidate populations produced by each algorithm at every iteration. For each pair of algorithms, on each problem instance and run, the test is applied to the two populations at the same iteration, and the proportion of iterations where the null hypothesis is not rejected is averaged across problems and runs to define a similarity score. These scores are then used to build a hierarchical clustering (Ward dendrogram) of 114 MEALPY algorithms evaluated on BBOB instances of dimension 2 and 5. The authors report that known variants of the same base algorithm cluster together, and they interpret this as evidence that the method can reveal similarities in search behavior that cut across metaphor-based groupings.
Significance. If the proposed method were statistically valid, it would directly address a recognized problem in the metaheuristics community: the proliferation of metaphor-based algorithms that are not positioned relative to existing methods. A quantitative, behavior-based similarity measure would be a useful diagnostic tool, and the paper is commendable for making its experimental code publicly available and for conducting a large-scale comparison on 114 algorithms. The idea of using multivariate distribution-free tests on search trajectories is promising. However, the central statistical claim is undermined by a fundamental violation of the independence assumption of the crossmatch test, so the reported similarity scores and the dendrogram are not trustworthy as presented. The paper's internal-consistency evidence (known variants clustering) is suggestive but not conclusive, and the lack of any uncertainty quantification further weakens the empirical claims.
major comments (2)
- [Section 3 (Methodology) and Section 4 (Experimental Design)] The paper states in Section 3 that "each pairwise comparison involves two independent samples," but the experimental design described in Section 4 and the algorithm-execution step in Section 3 share the same initial population and the same random seed across all algorithms. Therefore, the two populations compared at a fixed iteration and run are dependent: they start from an identical set of points and are driven by correlated randomness. The crossmatch test's exact null distribution (Rosenbaum 2005) is derived under the assumption of independent samples from the two distributions, and this independence is also required for the permutation-based p-value calculation. Since the samples are not independent, the reported p-values are not calibrated, and the similarity ratios and the Ward dendrogram in Figure 3 built from those p-values are statistically invalid. This is a load-bearing flaw: the entire contribution, the "novel empirical technique," rests on these p-values.
- [Section 3, "Empirical aggregation of outcomes" and Section 5] The similarity heuristic is defined as the mean ratio of iterations where the test fails to reject the null, but the paper provides no justification for this particular aggregation, no sensitivity analysis with respect to the significance threshold (0.05) or the Bonferroni correction (whose number of tests is not specified), and no error bars or confidence intervals for the similarity scores. Since the dendrogram in Figure 3 is a deterministic function of these point estimates, the robustness of the reported groupings is unknown. At minimum, the authors should show that the main clusters are stable under variations of the threshold and the aggregation rule.
minor comments (4)
- [Section 6 (Conclusion)] The word "empricial" should be "empirical."
- [Section 5, Figure 2 caption] The caption says "Values of the test statistic obtained with both tests," but only the crossmatch test is described; if a second test is used, it should be identified, otherwise the caption is misleading.
- [Section 5, Figure 3] The dendrogram is difficult to read because the algorithm labels are not legible at the printed scale; a zoomable or table-based listing of the clusters would be more informative.
- [Section 4 (Experimental Design)] The paper says the performance data was "utilized from a previous study" but does not specify which runs or seeds were used, nor how the shared initial population was constructed; a clearer data provenance description would help reproducibility.
Circularity Check
No circularity: the similarity scores are computed directly from algorithm trajectories, and the clustering validation is post-hoc, not an input to the derivation.
full rationale
The paper's derivation chain is empirical rather than deductive: it runs algorithms, computes crossmatch test statistics on their populations at each iteration, aggregates non-rejection rates into a similarity score, and clusters algorithms from that score. Nothing in this chain defines the target conclusion (that certain algorithms have similar search behavior) in terms of the conclusion itself, nor does it fit a parameter to the data and then rename that fit as a prediction. The known-variant groupings observed in the dendrogram are used as a sanity check after the scores are computed, which is an internal consistency check, not a load-bearing circular step. The one substantive concern raised by the reader's take is statistical: Section 3 states that each pairwise comparison involves two independent samples, while the experimental design shares initial populations and random seeds across algorithms, potentially violating the exact null distribution of the crossmatch test. That is a correctness or validity issue, not a circularity issue, because the reported p-values are not constructed to equal the conclusions; they are computed from the data and then interpreted. There is also no self-citation chain: the performance data comes from a previous study by different authors, and the crossmatch test is an external statistical method. Accordingly, no circular step can be exhibited, and the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption The two samples being compared at a fixed iteration and run are independent.
- domain assumption Min-max scaling of merged trajectories across all algorithms makes the populations comparable.
- domain assumption The mean percentage of non-rejected iterations across problems and runs is a valid similarity measure.
Cite this review
Pith. "Pith review of Comparing Optimization Algorithms Through the Lens of Search Behavior Analysis." pith.science (2026). https://pith.science/paper/SUTEFMIU
@misc{pith2026250701668,
author = {Pith},
title = {Pith review of: Comparing Optimization Algorithms Through the Lens of Search Behavior Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/SUTEFMIU}},
note = {Machine review of arXiv:2507.01668}
}
read the original abstract
The field of numerical optimization has recently seen a surge in the development of "novel" metaheuristic algorithms, inspired by metaphors derived from natural or human-made processes, which have been widely criticized for obscuring meaningful innovations and failing to distinguish themselves from existing approaches. Aiming to address these concerns, we investigate the applicability of statistical tests for comparing algorithms based on their search behavior. We utilize the cross-match statistical test to compare multivariate distributions and assess the solutions produced by 114 algorithms from the MEALPY library. These findings are incorporated into an empirical analysis aiming to identify algorithms with similar search behaviors.
Figures
Reference graph
Works this paper leans on
-
[1]
Claus Aranha, Christian L. Camacho Villalón, Felipe Campelo, Marco Dorigo, Rubén Ruiz, Marc Sevaux, Kenneth Sörensen, and Thomas Stützle. 2021. Metaphor-based metaheuristics, a call for action: the elephant in the room. Swarm Intelligence 16, 1 (Nov. 2021), 1–6. https://doi.org/10.1007/s11721-021-00202-9
-
[2]
Christian Leonardo Camacho-Villalón, Marco Dorigo, and Thomas Stützle. 2019. The intelligent water drops algorithm: why it cannot be considered a novel algorithm: A brief discussion on the use of metaphors in optimization. Swarm Intelligence 13, 3–4 (May 2019), 173–192. https://doi.org/10.1007/s11721-019-00165-y
-
[3]
Andrea De Lorenzo, Eric Medvet, Tea Tušar, and Alberto Bartoli. 2019. An analysis of dimensionality reduction techniques for visualizing evolution. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO ’19). Association for Computing Machinery, New York, NY, USA, 1864–1872. https://doi.org/10.1145/...
arXiv 2019
-
[4]
Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas Bäck. 2024. IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics. Evolutionary Computation 32, 3 (09 2024), 205–210. https://doi.org/10.1162/evco_a_00342
-
[5]
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633
work page 2009
-
[6]
Fionn Murtagh and Pierre Legendre. 2014. Ward’s hierarchical agglomerative clustering method: which algorithms implement Ward’s criterion? Journal of classification 31 (2014), 274–295
work page 2014
-
[7]
Paul Rosenbaum. 2005. An Exact Distribution-Free Test Comparing Two Multivariate Distributions Based on Adjacency. Journal of the Royal Statistical Society Series B 67 (09 2005), 515–530. https://doi.org/10.1111/j.1467-9868.2005.00513.x
arXiv 2005
-
[8]
Jörg Stork, Agoston E Eiben, and Thomas Bartz-Beielstein. 2022. A new taxonomy of global optimization algorithms.Natural Computing 21, 2 (2022), 219–242
work page 2022
Show all 12 references
-
[9]
Kenneth Sörensen. 2015. Metaheuristics—the metaphor exposed. International Transactions in Operational Research 22, 1 (2015), 3–18. https://doi.org/10.1111/itor.12001 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/itor.12001
2015 doi
-
[10]
Nguyen Van Thieu and Seyedali Mirjalili. 2023. MEALPY: An open-source library for latest meta-heuristic algorithms in Python.Journal of Systems Architecture 139 (2023), 102871. https://doi.org/10.1016/j.sysarc.2023.102871
2023
-
[11]
Kononova, and Thomas Bäck
Diederick Vermetten, Carola Doerr, Hao Wang, Anna V. Kononova, and Thomas Bäck. 2024. Large-Scale Benchmarking of Metaphor- Based Optimization Heuristics. In Proceedings of the Genetic and Evolutionary Computation Conference (Melbourne, VIC, Australia) (GECCO ’24). Association...
2024
-
[12]
Eric W Weisstein. 2004. Bonferroni correction. https://mathworld. wolfram. com/ (2004)
2004
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.