Pith. sign in

REVIEW 3 major objections 6 minor 42 references

A Standardized Benchmark Set of Clustering Problem Instances for Comparing Black-Box Optimizers

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A standardized suite of 40 clustering benchmark problems injects permutation symmetry and neutrality into black-box optimization evaluation.

desk verdict Useful standardized clustering benchmark with real gaps: neutrality is unquantified and internal diversity is weaker than claimed, but the resource is solid and deserves peer review. read the letter →

arxiv 2505.09233 v1 pith:6TYY6W44 submitted 2025-05-14 cs.NE

Diederick Vermetten , Catalin-Viorel Dinu , Marcus Gallagher This is my paper · ORCID
classification cs.NE MSC 90C2668T2090C59
keywords black-boxoptimizationbenchmarksuitedataclusteringpermutationinvarianceneutralityexploratorylandscapeanalysisCMA-ESprobleminstances
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that data-clustering problems, viewed as continuous black-box optimization over cluster-center coordinates, form a benchmark class that existing suites underrepresent because they are permutation-invariant and contain flat neutral regions. It builds a standardized set of 40 instances from 10 common datasets, each reduced to two dimensions with PCA and min-max normalized, with cluster counts $k \in \{2,3,5,10\}$, yielding problem dimensions 4, 6, 10, and 20. A study of 128 modular CMA-ES configurations plus exploratory landscape analysis is used to show that the instances behave similarly to each other but differently from the well-known BBOB suite, both in algorithm rankings and in landscape features. Local search and hill-valley analysis indicate genuine multimodality beyond the $k!$ symmetric copies, increasing with dimension. If the claims hold, the suite gives the optimization community a reusable open test bed for a class of real-world problems where the ordering of solutions does not matter, a class that standard benchmarks ignore.

What carries the argument

The central object is the MSE centroid clustering function $f(\mathbf{C}|X) = \frac{1}{n}\sum_{i=1}^n \min_j \lVert x_i - c_j \rVert^2$, with decision vector $\mathbf{y}=(\mathbf{c}_1,\dots,\mathbf{c}_k)\in\mathbb{R}^{kd}$. Because the assignment of data points to centers is by nearest neighbor, the landscape is piecewise quadratic with discontinuous ridges where assignment changes; because the ordering of centers is arbitrary, every solution has $k!$ permuted copies; and because a center can be nearest to no point, large perfectly flat regions exist outside the data range. The paper's standardization machinery fixes one family of such problems by PCA-reducing each dataset to two dimensions, min-max scaling, and choosing cluster counts; its analysis machinery is a portfolio of 128 CMA-ES configurations ranked per problem, exploratory landscape analysis features computed from 4096 Sobol samples, local searches combined with hill-valley cliques to count distinct basins, and a stick-breaking map that orders centers to remove the permutation symmetry.

What would settle it

Take one of the 10 datasets, compute the original high-dimensional clustering landscape, then compare its number of distinct local basins and its ELA feature vector to the PCA-reduced $[0,1]^2$ version used in the suite; if the reduced problem has fewer basins or a markedly different ELA profile, the standardization step is not neutral. A sharper test: construct a synthetic dataset with three well-separated high-dimensional clusters engineered so that PCA projects two clusters together, and check whether the projected problem still exhibits the same $k!$ symmetric structure and neutrality as the original.

Watch

Extended reading notes

Core claim

The central claim is that the mean-squared-error centroid clustering objective, with the decision vector being the coordinates of $k$ cluster centers, induces continuous landscapes whose defining characteristics are $k!$ symmetric basins and plateaus where a center is not the closest to any data point, and that these characteristics can be standardized into a benchmark suite without losing their challenge. The paper operationalizes this by selecting 10 datasets, projecting each to two dimensions with PCA, min-max normalizing to $[0,1]^2$, and fixing $k\in\{2,3,5,10\}$; the result is 40 problems of dimension 4, 6, 10, and 20. Its experiments show that CMA-ES configuration rankings are highly correlated within the suite (Kendall's $\tau$ around 0.8–0.9) but much less correlated with BBOB problems, and that exploratory landscape features also separate the clustering problems from most BBOB functions. Local searches from 50 starts reveal that the fraction of runs staying in their initial symmetry region drops from roughly 75% at dimension 4 to near zero at dimension 20, and clique-based analysis finds multiple distinct basins, establishing multimodality beyond symmetry. The paper also tests a stick-breaking transformation meant to map the search space to one symmetry region and finds it does not improve default CMA-ES performance, sometimes making it worse.

Load-bearing premise

The load-bearing premise is that PCA-reducing every dataset to two dimensions and min-max normalizing it preserves the landscape features—permutation symmetry, neutrality, and multimodality—that make clustering problems interesting; if the projection distorts or destroys those features, the suite may no longer represent the class it claims to benchmark.

Editorial extensions

If this is right

  • Algorithms compared on this suite are being tested on a landscape property—permutation invariance with neutrality—that the BBOB suite does not isolate, so rankings from this suite complement rather than repeat standard benchmark results.
  • The high internal rank correlation (around 0.8–0.9) means the 10 datasets behave like a coherent problem class, so a configuration chosen on one is likely to generalize to the others at the same dimension.
  • Because the number of distinct basins grows with $k$, higher-dimensional instances in the suite will separate algorithms by their ability to escape local minima, not just by convergence speed.
  • The stick-breaking transformation does not help default CMA-ES and sometimes hurts, implying that simply removing symmetry is not enough; the induced hierarchical dependence between variables becomes a new obstacle.
  • The K-Means++ baselines give a cheap external reference, letting black-box optimizers be compared directly against a specialized non-black-box clustering method on the same objective values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit is to benchmark symmetry-aware optimizers, such as those that sort or canonicalize cluster centers, on this suite; the stick-breaking result suggests such methods must avoid introducing variable hierarchies.
  • Because PCA reduction is linear and global, datasets whose clusters are separable only in higher dimensions may lose their cluster structure in the projected instances; generating synthetic datasets with known high-dimensional separation and comparing landscape features before and after reduction would test this.
  • The neutrality regions should expand as $k$ grows, since more centers are likely to be unused; this predicts that algorithms with explicit plateau handling or restarts will improve relative to standard CMA-ES on larger $k$, a hypothesis the paper does not test.
  • The generator's freedom to change distance and error metrics invites a landscape-sensitivity study: replacing squared distance with a max-error measure would likely shrink the neutral regions, isolating how much of the suite's challenge comes from neutrality versus symmetry.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces IOHclustering, a benchmark suite of 40 continuous black-box optimization problem instances derived from 10 real-world datasets reduced to 2D by PCA and min-max normalized, with k in {2,3,5,10} cluster centers yielding search dimensions 4, 6, 10, and 20. The authors argue that clustering landscapes exhibit permutation invariance and neutrality, properties that are underrepresented in existing suites, and they integrate the suite with the IOHprofiler framework. To validate the suite, they benchmark 128 modular CMA-ES configurations with 25 repetitions each, compare algorithm rankings and ELA features against BBOB, analyze multimodality through local search and hill-valley tests, and test a symmetry-breaking transformation. The central claim is that the suite is a reusable, standardized resource for benchmarking black-box optimizers on problems with permutation symmetry and neutrality.

Significance. The main contribution is a resource rather than a new algorithm, and, conditional on the design choices being accepted, it is a useful one. The paper is strongest on practical infrastructure: the suite is open-source, integrated with IOHprofiler and IOHanalyzer, ships with a problem generator, provides K-Means++ baselines for every instance, and the entire benchmarking study is reproducible through a Zenodo repository with fixed seeds and 25 repetitions. The experimental execution is careful and transparent. I found no circularity in the evaluation: the K-Means++ baselines are computed on the suite but are not used to fit algorithm parameters, and the CMA-ES portfolio is used only for characterization. The main risk is that two of the suite's advertised properties, neutrality and diversity, are not convincingly demonstrated for the standardized instances, and the PCA preprocessing is not validated as preserving the landscape properties that motivate the suite. These issues are fixable with additional analysis, which is why I recommend major revision rather than rejection.

major comments (3)
  1. [Section 2.3 / Section 2.2] The neutrality property advertised in the abstract and Section 2.2 is never quantified for the 40 standardized instances. The only illustration of neutrality (Figure 1, right-hand plots) uses the original unconstrained formulation with an enlarged search domain; after min-max normalization and the box constraint [0,1]^m introduced in Section 2.3, the 'outside the data range' flat regions are excluded, and neutrality can only occur for centers that lie inside the box but are not the closest center of any data point. The paper reports no measurement of the existence or volume of such regions for any of the 40 instances. Please add a quantitative neutrality analysis (e.g., Monte Carlo volume estimates or a neutrality metric per instance) or explicitly qualify the claim to 'potential regions of neutrality' consistent with the evidence presented.
  2. [Section 3.1 / Section 3.2, Figures 5, 6, 8a] The evidence presented undercuts the diversity criterion from [3] that the paper adopts. Kendall's tau between the 10 clustering problems in Figure 8a is 0.7-0.9, and Figure 6 is described in Section 3.1 as showing that 'overall differences between datasets are relatively small'. These results suggest that the 40 instances are highly redundant from the algorithm-ranking perspective, which weakens the claim that the suite provides a diverse set of benchmarking challenges. The ELA cosine similarities in Figure 8b show more spread, but the paper does not reconcile this with the high rank correlations. Please discuss this tension explicitly and either select or generate instances with lower internal correlation, or restrict the diversity claim to landscape features rather than algorithm performance.
  3. [Section 2.3] The reduction of every dataset to 2D by PCA and subsequent min-max normalization is not validated as preserving the landscape properties that motivate the suite. The clustering objective is permutation-invariant in any dimension, but the PCA projection can change the distance geometry, cluster separability, and the size and location of neutral regions, and the min-max scaling can introduce additional artifacts. Since these transformations are applied before the landscape properties are analyzed, the representativeness of the resulting instances relative to real clustering problems is an assumption rather than a demonstrated fact. Please provide a validation for at least a subset of datasets, for example by comparing ELA features, neutral-region sizes, or algorithm rankings between the original high-dimensional datasets and their 2D projections.
minor comments (6)
  1. [Equations (1)-(3), Section 2.3] The symbol n is used both for the number of data points (Equation 1) and for the problem dimensionality ('dimensionalities n in {4,6,10,20}'); please disambiguate the notation.
  2. [Figure 3] The y-axis label 'value' is not informative; state that the plotted quantity is the MSE objective value, and fix the spacing in 'T rue' in the legend.
  3. [Section 2.2] There is a typo: 'therefor' should be 'therefore', and the phrase '(i.e. non non-equidistant)' contains a doubled 'non'.
  4. [Section 3.2] The comparison to BBOB uses only the first instance of each BBOB function; because BBOB defines multiple instances with different translations and rotations, this choice should be stated as a limitation or justified in the text.
  5. [Section 3.4] The parameters of the Beta distribution are not defined: 'X' is used for the dataset in Equation (1), but in the transformation it appears to denote a different quantity (apparently the number of components), and the inverse-CDF notation should be explained.
  6. [Figure 7 caption] The caption says 'Best MSE (*100)' while the text describes differences to the minimal value found by K-Means++; please align the caption with the quantity actually plotted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction, K-Means++ baselines, and BBOB comparisons do not define their outputs in terms of their inputs.

full rationale

No circular reduction is present in the paper. The benchmark suite is constructed directly from the clustering objective (Eq. 1) with fixed preprocessing (PCA and min-max normalization, Sec. 2.3), and no parameter is fitted to benchmark results and then reported as a prediction. The K-Means++ values are external baselines used only for comparison in Figures 3, 4, and 7, so they are not fitted inputs being renamed as predictions. The CMA-ES and ELA comparisons against BBOB are matched experimental protocols rather than self-confirming arguments: the claim that clustering problems differ from BBOB is supported by external ELA feature sets and by ranking correlations computed from independent runs. The paper's self-citations ([13], [33], [9], [10], [40], and related tool papers) are used as motivation, prior problem instances, or software infrastructure; none is invoked as a uniqueness theorem or as the sole support for a central conclusion. One caveat is that the neutrality property is argued qualitatively in Sec. 2.2 and used as motivation for the suite, but its extent is not quantified for the 40 standardized instances in Sec. 2.3; this is a support gap that belongs under correctness or completeness risk, not circularity, because no benchmark result is defined in terms of the neutrality claim. The paper is therefore self-contained in its main benchmarking claims and receives a score of 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark suite involves design choices (k values, PCA dimension, normalization) rather than fitted parameters. The central claim relies on domain assumptions about the representativeness of the constructed instances and the validity of the analysis tools.

free parameters (3)
  • number of clusters k = {2, 3, 5, 10}
    Design choice for the benchmark suite to yield search dimensions 4, 6, 10, 20; not fitted to data.
  • PCA target dimensionality = 2
    Chosen to standardize dimensionality across datasets; a modeling choice that shapes the suite.
  • min-max normalization bounds = [0, 1]
    Applied to the data-space to provide consistent search domains; changes the landscape.
assumptions (3)
  • domain assumption PCA preserves the optimization-relevant structure of the clustering datasets.
    The paper assumes that 2D PCA projections retain the landscape properties (symmetry, multimodality) that make clustering problems interesting benchmarks; Section 2.3.
  • domain assumption K-Means++ provides a strong baseline for solution quality.
    Used as baseline for EAF bounds and performance comparison; Section 3.
  • domain assumption ELA features computed with PFlacco and cosine similarity capture problem similarity.
    The comparison to BBOB rests on this; Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Standardized Benchmark Set of Clustering Problem Instances for Comparing Black-Box Optimizers." pith.science (2026). https://pith.science/paper/6TYY6W44

@misc{pith2026250509233,
  author       = {Pith},
  title        = {Pith review of: A Standardized Benchmark Set of Clustering Problem Instances for Comparing Black-Box Optimizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TYY6W44}},
  note         = {Machine review of arXiv:2505.09233}
}
read the original abstract

One key challenge in optimization is the selection of a suitable set of benchmark problems. A common goal is to find functions which are representative of a class of real-world optimization problems in order to ensure findings on the benchmarks will translate to relevant problem domains. While some problem characteristics are well-covered by popular benchmarking suites, others are often overlooked. One example of such a problem characteristic is permutation invariance, where the search space consists of a set of symmetrical search regions. This type of problem occurs e.g. when a set of solutions has to be found, but the ordering within this set does not matter. The data clustering problem, often seen in machine learning contexts, is a clear example of such an optimization landscape, and has thus been proposed as a base from which optimization benchmarks can be created. In addition to the symmetry aspect, these clustering problems also contain potential regions of neutrality, which can provide an additional challenge to optimization algorithms. In this paper, we present a standardized benchmark suite for the evaluation of continuous black-box optimization algorithms, based on data clustering problems. To gain insight into the diversity of the benchmark set, both internally and in comparison to existing suites, we perform a benchmarking study of a set of modular CMA-ES configurations, as well as an analysis using exploratory landscape analysis. Our benchmark set is open-source and integrated with the IOHprofiler benchmarking framework to encourage its use in future research.

Figures

Figures reproduced from arXiv: 2505.09233 by the authors.

Figure 1
Figure 1. Contour plots of selected clustering problems based on 1-dimensional data (visualized as red crosses below the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data underlying each of the 10 problem sets. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Some example convergence curves (geometric mean) for different population sizes and elitism options (bound [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: EAF over all 10-dimensional problems in our suite, [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Distribution of parameters in the top 8 (out of 128) [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: For each problem, the performance (average final [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Pairwise similarities between the clustering and BBOB problems. Shown here for dimensionality 10, other dimension [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Fraction of solutions that ended up in the same [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Number of distinct local minima discovered by local search algorithms across different problem instances. The [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Example of Original and Transformed version of [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Differences in mean performance of CMA-ES on the original vs transformed space. Negative values corre￾spond to functions where the original representation led to better performance. We can see from [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 33 canonical work pages

  1. [3]

    Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel Lopez-Ibanez, et al. 2020. Benchmarking in optimiza- tion: Best practice and open issues. arXiv preprint arXiv:2007.03488 (2020)

    arXiv 2020

  2. [1]

    2006.k-means++: The advantages of careful seeding

    David Arthur and Sergei Vassilvitskii. 2006.k-means++: The advantages of careful seeding. Technical Report. Stanford

    2006

  3. [2]

    Adil M Bagirov. 2008. Modified global k-means algorithm for minimum sum-of- squares clustering problems. Pattern Recognition 41, 10 (2008), 3192–3199

    work page 2008

  4. [4]

    Pauline Bennet, Carola Doerr, Antoine Moreau, Jeremy Rapin, Fabien Teytaud, and Olivier Teytaud. 2021. Nevergrad: black-box optimization platform. ACM SIGEVOlution 14, 1 (2021), 8–15

    work page 2021

  5. [5]

    Laurens Bliek, Arthur Guijt, Rickard Karlsson, Sicco Verwer, and Mathijs De Weerdt. 2023. Benchmarking surrogate-based optimisation algorithms on expensive black-box functions. Applied Soft Computing 147 (2023), 110744

    work page 2023

  6. [6]

    Jack Brimberg, Pierre Hansen, N Mladenovic, and Said Salhi. 2008. A survey of solution methods for the continuous location-allocation problem. International Journal of Operations Research 5, 1 (2008), 1–12

    work page 2008

  7. [7]

    Theodore Brown, Alexandru Cioba, and Ilija Bogunovic. 2024. Sample-efficient bayesian optimisation using known invariances. Advances in Neural Information Processing Systems 37 (2024), 47931–47965

    work page 2024

  8. [8]

    An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. 1993. On the geometry of feedforward neural network error surfaces. Neural computation 5, 6 (1993), 910–927

    work page 1993

Show all 42 references
  1. [9]

    Jacob de Nobel, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas Bäck. 2021. Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1375–1384

    2021

  2. [10]

    Jacob de Nobel, Furong Ye, Diederick Vermetten, Hao Wang, Carola Doerr, and Thomas Bäck. 2024. Iohexperimenter: Benchmarking platform for iterative optimization heuristics. Evolutionary Computation 32, 3 (2024), 205–210

    2024

  3. [11]

    Sofianos Panagiotis Fotias, Ismail Ismail, and Vassilis Gaganis. 2024. Optimization of Well Placement in Carbon Capture and Storage (CCS): Bayesian Optimization Diederick Vermetten, Catalin-Viorel Dinu, and Marcus Gallagher Framework under Permutation Invariance. Applied Scien...

    2024

  4. [12]

    Pasi Fränti and Sami Sieranoja. 2018. K-means properties on six clustering benchmark datasets. , 4743–4759 pages. http://cs.uef.fi/sipu/datasets/

    2018

  5. [13]

    Marcus Gallagher. 2016. Towards improved benchmarking of black-box op- timization algorithms using clustering problems. Soft Computing 20 (2016), 3835–3849

    2016

  6. [14]

    Marcus Gallagher. 2019. Fitness landscape analysis in data-driven optimization: An investigation of clustering problems. In 2019 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2308–2314

    2019

  7. [15]

    Sara Hajari and Marcus Gallagher. 2024. Searching for Benchmark Problem Instances from Data-Driven Optimisation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion . 139–142

    2024

  8. [16]

    Nikolaus Hansen. 2008. Adaptive encoding: How to render search coordinate system invariant. In International Conference on Parallel Problem Solving from Nature. Springer, 205–214

    2008

  9. [17]

    Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2021. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software 36, 1 (2021), 114–144

    2021

  10. [18]

    Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real- Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Defi- nitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633/ document

    2009

  11. [19]

    Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9, 2 (2001), 159–195

    2001

  12. [20]

    Robert Hecht-Nielsen. 1989. Neurocomputing. Addison-Wesley

    1989

  13. [21]

    Anil K Jain, M Narasimha Murty, and Patrick J Flynn. 1999. Data clustering: a review. ACM computing surveys (CSUR) 31, 3 (1999), 264–323

    1999

  14. [22]

    Maurice George Kendall. 1948. Rank correlation methods. (1948)

    1948

  15. [23]

    Pascal Kerschke, Marcus Gallagher, Mike Preuss, and Olivier Teytaud. 2019. The Machine Learning and Data Analysis (MLDA) Problem Set, v1.0. (2019). Part of GECCO workshop UMLOP

    2019

  16. [24]

    Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. 2022. Learning the characteristics of engineering optimization problems with applications in automotive crash. In Proceedings of the Genetic and Evolutionary Computation Conference . 1227–1236

    2022

  17. [25]

    Manuel López-Ibáñez, Juergen Branke, and Luís Paquete. 2021. Reproducibility in evolutionary computation. ACM Transactions on Evolutionary Learning and Optimization 1, 4 (2021), 1–21

    2021

  18. [26]

    Manuel López-Ibáñez, Diederick Vermetten, Johann Dreo, and Carola Doerr. 2024. Using the empirical attainment function for analyzing single-objective black-box optimization algorithms. IEEE Transactions on Evolutionary Computation (2024)

    2024

  19. [27]

    Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory landscape analysis. In Proceedings of the 13th annual conference on Genetic and evolutionary computation . 829–836

    2011

  20. [28]

    Kevin P. Murphy. 2022. Probabilistic Machine Learning: An introduction . MIT Press. http://probml.github.io/book1

    2022

  21. [29]

    Daiki Otaki, Hirofumi Nonaka, and Noboru Yamada. 2022. Thermal design optimization of electronic circuit board layout with transient heating chips by using Bayesian optimization and thermal network model. International Journal of Heat and Mass Transfer 184 (2022), 122263

    2022

  22. [30]

    Michael JD Powell. 1964. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The computer journal 7, 2 (1964), 155–162

    1964

  23. [31]

    Raphael Patrick Prager and Heike Trautmann. 2023. Nullifying the inherent bias of non-invariant exploratory landscape analysis features. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar) . Springer, 411–425

    2023

  24. [32]

    Raphael Patrick Prager and Heike Trautmann. 2024. Pflacco: Feature-based landscape analysis of continuous and constrained optimization problems in Python. Evolutionary Computation 32, 3 (2024), 211–216

    2024

  25. [33]

    Jérémy Rapin, Marcus Gallagher, Pascal Kerschke, Mike Preuss, and Olivier Teytaud. 2019. Exploring the MLDA benchmark on the nevergrad platform. In Proceedings of the Genetic and Evolutionary Computation Conference Companion . 1888–1896

    2019

  26. [34]

    Matthew Stephens. 2000. Dealing with label switching in mixture models.Journal of the Royal Statistical Society: Series B (Statistical Methodology) 62, 4 (2000), 795– 809

    2000

  27. [35]

    Mohammad-H Tayarani-N and Adam Prügel-Bennett. 2013. On the landscape of combinatorial optimization problems. IEEE Transactions on Evolutionary Computation 18, 3 (2013), 420–434

    2013

  28. [36]

    Sébastien Verel, Gabriela Ochoa, and Marco Tomassini. 2010. Local optima networks of NK landscapes with neutrality. IEEE Transactions on Evolutionary Computation 15, 6 (2010), 783–797

    2010

  29. [37]

    Diederick Vermetten, Catalin-Viorel Dinu, and Marcus Gallagher. 2025. Repro- ducibility and additional files. Zenodo (2025). https://doi.org/10.5281/zenodo. 15302416

    2025 doi

  30. [38]

    Diederick Vermetten, Jeroen Rook, Oliver L Preuß, Jacob de Nobel, Carola Do- err, Manuel López-Ibañez, Heike Trautmann, and Thomas Bäck. 2025. MO- IOHinspector: Anytime Benchmarking of Multi-Objective Algorithms using IOHprofiler. In International Conference on Evolutionary Mu...

    2025

  31. [39]

    Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al . 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature methods 17, 3 (2020), 261–272

    2020

  32. [40]

    Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, and Thomas Bäck

  33. [41]

    Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. 1997. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on mathematical software (TOMS) 23, 4 (1997), 550–560

    1997

  34. [2022]

    ACM Transactions on Evolutionary Learning and Optimization 2, 1 (2022), 1–29

    IOHanalyzer: Detailed performance analyses for iterative optimization heuristics. ACM Transactions on Evolutionary Learning and Optimization 2, 1 (2022), 1–29

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.