{"id":"a3afa372-03a5-4520-b2e1-a551df78b495","arxiv_id":"2505.09233","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A new open-source benchmark suite uses PCA-reduced clustering datasets to create continuous optimization problems with permutation symmetry and neutrality.","lead":"This paper introduces a standardized benchmark suite of 40 clustering problem instances for continuous black-box optimization, built from 10 datasets with 2 to 10 cluster centers. The suite is integrated with the IOHprofiler framework and comes with code, data, and baselines, plus analyses comparing it to the BBOB suite.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Neutrality is advertised but never quantified for the 40 standardized instances; PCA plus box constraints may leave negligible neutral volume, so a core claimed feature is unverified.","rationale":"Reading the paper in good faith, the resource is open, reproducible, and the symmetry property is rigorous. The main contribution does not depend on proving difficult theorems, so I did not look for a formal flaw. The weakest point is the empirical claim that the suite contains neutral regions. The reader's PCA concern is broader and related; my attack isolates the specific mechanism by which PCA plus the [0,1]^m box could erase neutrality. This is load-bearing because the abstract and introduction advertise neutrality as a second challenge beyond permutation invariance. The Monte Carlo audit directly measures the property; it is a validation task the authors could add with their existing package. If the audit shows substantial neutral volume, my concern is resolved and the paper can be accepted with minor additions; if not, the claim needs to be qualified or the suite redesigned. Either way the reader's CONDITIONAL verdict is appropriate, so I do not request a verdict change.","tokens_in":24131,"tokens_out":10544,"duration_ms":118670,"concrete_test":"Neutrality audit of the released IOHclustering instances: for each of the 40 problems, sample 10^6 points uniformly in [0,1]^m, evaluate f from Eq. (1) with the package's assignment rule, and record whether any cluster center has b_i,j = 0 for all data points. Report the fraction of samples with at least one inactive center and mean inactive-center fraction per problem. If the neutral fraction is below 10^-3 for a majority of instances, the neutrality component of the central claim is not supported; if it is substantial, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the suite exposes optimizers to neutral landscapes is the least structurally guaranteed part of the paper. Permutation invariance follows automatically from the ordering of cluster centers in y (Sec. 2.1), but neutrality is only demonstrated for a one-dimensional toy problem with an enlarged search domain (Fig. 1, right plots). In the standardized instances (Sec. 2.3), every dataset is min-max normalized to [0,1]^2 while each center's variables are constrained to [0,1]^2 by the box [0,1]^m; the 'outside the data range' neutrality of Sec. 2.2 can therefore only occur when a center is inside [0,1]^2 yet is not the closest center to any data point. Whether such regions exist, and how large they are, is never measured for the 40 instances. If their volume is negligible for most instances, one of the two advertised problem characteristics is effectively absent, and the suite's claimed contribution over existing benchmarks is reduced to symmetry alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IOHclustering, a benchmark suite of 40 continuous black-box optimization problem instances derived from 10 real-world datasets reduced to 2D by PCA and min-max normalized, with k in {2,3,5,10} cluster centers yielding search dimensions 4, 6, 10, and 20. The authors argue that clustering landscapes exhibit permutation invariance and neutrality, properties that are underrepresented in existing suites, and they integrate the suite with the IOHprofiler framework. To validate the suite, they benchmark 128 modular CMA-ES configurations with 25 repetitions each, compare algorithm rankings and ELA features against BBOB, analyze multimodality through local search and hill-valley tests, and test a symmetry-breaking transformation. The central claim is that the suite is a reusable, standardized resource for benchmarking black-box optimizers on problems with permutation symmetry and neutrality.","tokens_in":24340,"tokens_out":5814,"duration_ms":58169,"significance":"The main contribution is a resource rather than a new algorithm, and, conditional on the design choices being accepted, it is a useful one. The paper is strongest on practical infrastructure: the suite is open-source, integrated with IOHprofiler and IOHanalyzer, ships with a problem generator, provides K-Means++ baselines for every instance, and the entire benchmarking study is reproducible through a Zenodo repository with fixed seeds and 25 repetitions. The experimental execution is careful and transparent. I found no circularity in the evaluation: the K-Means++ baselines are computed on the suite but are not used to fit algorithm parameters, and the CMA-ES portfolio is used only for characterization. The main risk is that two of the suite's advertised properties, neutrality and diversity, are not convincingly demonstrated for the standardized instances, and the PCA preprocessing is not validated as preserving the landscape properties that motivate the suite. These issues are fixable with additional analysis, which is why I recommend major revision rather than rejection.","major_comments":[{"comment":"The neutrality property advertised in the abstract and Section 2.2 is never quantified for the 40 standardized instances. The only illustration of neutrality (Figure 1, right-hand plots) uses the original unconstrained formulation with an enlarged search domain; after min-max normalization and the box constraint [0,1]^m introduced in Section 2.3, the 'outside the data range' flat regions are excluded, and neutrality can only occur for centers that lie inside the box but are not the closest center of any data point. The paper reports no measurement of the existence or volume of such regions for any of the 40 instances. Please add a quantitative neutrality analysis (e.g., Monte Carlo volume estimates or a neutrality metric per instance) or explicitly qualify the claim to 'potential regions of neutrality' consistent with the evidence presented.","section":"Section 2.3 / Section 2.2"},{"comment":"The evidence presented undercuts the diversity criterion from [3] that the paper adopts. Kendall's tau between the 10 clustering problems in Figure 8a is 0.7-0.9, and Figure 6 is described in Section 3.1 as showing that 'overall differences between datasets are relatively small'. These results suggest that the 40 instances are highly redundant from the algorithm-ranking perspective, which weakens the claim that the suite provides a diverse set of benchmarking challenges. The ELA cosine similarities in Figure 8b show more spread, but the paper does not reconcile this with the high rank correlations. Please discuss this tension explicitly and either select or generate instances with lower internal correlation, or restrict the diversity claim to landscape features rather than algorithm performance.","section":"Section 3.1 / Section 3.2, Figures 5, 6, 8a"},{"comment":"The reduction of every dataset to 2D by PCA and subsequent min-max normalization is not validated as preserving the landscape properties that motivate the suite. The clustering objective is permutation-invariant in any dimension, but the PCA projection can change the distance geometry, cluster separability, and the size and location of neutral regions, and the min-max scaling can introduce additional artifacts. Since these transformations are applied before the landscape properties are analyzed, the representativeness of the resulting instances relative to real clustering problems is an assumption rather than a demonstrated fact. Please provide a validation for at least a subset of datasets, for example by comparing ELA features, neutral-region sizes, or algorithm rankings between the original high-dimensional datasets and their 2D projections.","section":"Section 2.3"}],"minor_comments":[{"comment":"The symbol n is used both for the number of data points (Equation 1) and for the problem dimensionality ('dimensionalities n in {4,6,10,20}'); please disambiguate the notation.","section":"Equations (1)-(3), Section 2.3"},{"comment":"The y-axis label 'value' is not informative; state that the plotted quantity is the MSE objective value, and fix the spacing in 'T rue' in the legend.","section":"Figure 3"},{"comment":"There is a typo: 'therefor' should be 'therefore', and the phrase '(i.e. non non-equidistant)' contains a doubled 'non'.","section":"Section 2.2"},{"comment":"The comparison to BBOB uses only the first instance of each BBOB function; because BBOB defines multiple instances with different translations and rotations, this choice should be stated as a limitation or justified in the text.","section":"Section 3.2"},{"comment":"The parameters of the Beta distribution are not defined: 'X' is used for the dataset in Equation (1), but in the transformation it appears to denote a different quantity (apparently the number of components), and the inverse-CDF notation should be explained.","section":"Section 3.4"},{"comment":"The caption says 'Best MSE (*100)' while the text describes differences to the minimal value found by K-Means++; please align the caption with the quantity actually plotted.","section":"Figure 7 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a solid resource paper with careful experimental execution and good reproducibility practices. My main concern is that the two headline properties, neutrality and diversity, are asserted rather than demonstrated for the actual standardized instances, and the PCA preprocessing is not validated. These are fixable with additional analysis rather than fatal flaws, so major revision is appropriate. I would also encourage the editor to confirm that the Zenodo repository contains all promised data and scripts, since reproducibility is a central selling point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Vermetten, Dinu, and Gallagher have put together a genuinely useful resource: a standardized, IOHprofiler-integrated benchmark suite of 40 clustering instances with K-Means++ baselines, full reproducibility artifacts, and a careful comparison against BBOB. The paper is worth reading for anyone doing black-box optimization benchmarking. But the two headline properties—neutrality and diversity—are not as firmly established as the abstract suggests. The neutrality of the standardized instances is never quantified; the demonstration of flat regions comes from a toy problem with an expanded domain, while the actual suite restricts centers to [0,1]^2 after PCA and min-max normalization, where neutral volume may be negligible. This is a real gap, not a nitpick: if the 40 instances have almost no neutral regions, the suite is effectively just permutation-symmetric, and the claimed advantage over existing benchmarks shrinks. A simple measurement of the relative volume of regions where a center is inside the box but not nearest to any data point would settle it. The diversity claim is also softer than the text implies. Internal Kendall tau values among clustering instances are 0.8-0.9, meaning the 128 CMA-ES configs rank nearly identically across most of the suite. The ELA analysis shows more spread, but that doesn't erase the algorithmic redundancy. The authors acknowledge the high similarity but don't grapple with what it means for the suite's stated goal of providing diverse challenges. On the positive side, the PCA reduction to 2D is the right call for fixed dimensionality, but it is not validated against the original higher-dimensional data; some evidence that the reduced instances retain clustering structure (e.g., comparing landscape features or optimizer rankings) would strengthen the construction. I don't think any of these issues are fatal. The suite is usable, the code and data are there, the K-Means++ baselines are a nice touch, and the comparison methodology is solid. The paper deserves a serious refereeing; the fixes are concrete and mostly additional analysis, not rework. I'd send it to review with a request for neutral-region measurement and a diversity assessment that goes beyond average rank correlations. Who is this for? Benchmarking researchers and anyone comparing continuous optimizers on problems with symmetries. Not a landmark, but a legitimate contribution that will probably get cited.","headline":"Useful standardized clustering benchmark with real gaps: neutrality is unquantified and internal diversity is weaker than claimed, but the resource is solid and deserves peer review.","tokens_in":24823,"tokens_out":2611,"would_cite":true,"duration_ms":26542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C26","68T20","90C59"],"pacs":[],"model":"deepseek-v4-flash","headline":"A standardized suite of 40 clustering benchmark problems injects permutation symmetry and neutrality into black-box optimization evaluation.","keywords":["black-box optimization","benchmark suite","data clustering","permutation invariance","neutrality","exploratory landscape analysis","CMA-ES","clustering problem instances"],"falsifier":"Take one of the 10 datasets, compute the original high-dimensional clustering landscape, then compare its number of distinct local basins and its ELA feature vector to the PCA-reduced $[0,1]^2$ version used in the suite; if the reduced problem has fewer basins or a markedly different ELA profile, the standardization step is not neutral. A sharper test: construct a synthetic dataset with three well-separated high-dimensional clusters engineered so that PCA projects two clusters together, and check whether the projected problem still exhibits the same $k!$ symmetric structure and neutrality as the original.","tokens_in":23956,"feed_emoji":"📊","tokens_out":9068,"duration_ms":85298,"temperature":0.7,"pith_summary":"The paper proposes that data-clustering problems, viewed as continuous black-box optimization over cluster-center coordinates, form a benchmark class that existing suites underrepresent because they are permutation-invariant and contain flat neutral regions. It builds a standardized set of 40 instances from 10 common datasets, each reduced to two dimensions with PCA and min-max normalized, with cluster counts $k \\in \\{2,3,5,10\\}$, yielding problem dimensions 4, 6, 10, and 20. A study of 128 modular CMA-ES configurations plus exploratory landscape analysis is used to show that the instances behave similarly to each other but differently from the well-known BBOB suite, both in algorithm rankings and in landscape features. Local search and hill-valley analysis indicate genuine multimodality beyond the $k!$ symmetric copies, increasing with dimension. If the claims hold, the suite gives the optimization community a reusable open test bed for a class of real-world problems where the ordering of solutions does not matter, a class that standard benchmarks ignore.","feed_headline":"40 clustering problems test optimizers on symmetry and flat spots","feed_subtitle":"A new open suite shows optimizer rankings that BBOB misses, with k! symmetric regions and neutral plateaus.","key_machinery":"The central object is the MSE centroid clustering function $f(\\mathbf{C}|X) = \\frac{1}{n}\\sum_{i=1}^n \\min_j \\lVert x_i - c_j \\rVert^2$, with decision vector $\\mathbf{y}=(\\mathbf{c}_1,\\dots,\\mathbf{c}_k)\\in\\mathbb{R}^{kd}$. Because the assignment of data points to centers is by nearest neighbor, the landscape is piecewise quadratic with discontinuous ridges where assignment changes; because the ordering of centers is arbitrary, every solution has $k!$ permuted copies; and because a center can be nearest to no point, large perfectly flat regions exist outside the data range. The paper's standardization machinery fixes one family of such problems by PCA-reducing each dataset to two dimensions, min-max scaling, and choosing cluster counts; its analysis machinery is a portfolio of 128 CMA-ES configurations ranked per problem, exploratory landscape analysis features computed from 4096 Sobol samples, local searches combined with hill-valley cliques to count distinct basins, and a stick-breaking map that orders centers to remove the permutation symmetry.","core_discovery":"The central claim is that the mean-squared-error centroid clustering objective, with the decision vector being the coordinates of $k$ cluster centers, induces continuous landscapes whose defining characteristics are $k!$ symmetric basins and plateaus where a center is not the closest to any data point, and that these characteristics can be standardized into a benchmark suite without losing their challenge. The paper operationalizes this by selecting 10 datasets, projecting each to two dimensions with PCA, min-max normalizing to $[0,1]^2$, and fixing $k\\in\\{2,3,5,10\\}$; the result is 40 problems of dimension 4, 6, 10, and 20. Its experiments show that CMA-ES configuration rankings are highly correlated within the suite (Kendall's $\\tau$ around 0.8–0.9) but much less correlated with BBOB problems, and that exploratory landscape features also separate the clustering problems from most BBOB functions. Local searches from 50 starts reveal that the fraction of runs staying in their initial symmetry region drops from roughly 75% at dimension 4 to near zero at dimension 20, and clique-based analysis finds multiple distinct basins, establishing multimodality beyond symmetry. The paper also tests a stick-breaking transformation meant to map the search space to one symmetry region and finds it does not improve default CMA-ES performance, sometimes making it worse.","pith_inferences":["A direct extension the paper leaves implicit is to benchmark symmetry-aware optimizers, such as those that sort or canonicalize cluster centers, on this suite; the stick-breaking result suggests such methods must avoid introducing variable hierarchies.","Because PCA reduction is linear and global, datasets whose clusters are separable only in higher dimensions may lose their cluster structure in the projected instances; generating synthetic datasets with known high-dimensional separation and comparing landscape features before and after reduction would test this.","The neutrality regions should expand as $k$ grows, since more centers are likely to be unused; this predicts that algorithms with explicit plateau handling or restarts will improve relative to standard CMA-ES on larger $k$, a hypothesis the paper does not test.","The generator's freedom to change distance and error metrics invites a landscape-sensitivity study: replacing squared distance with a max-error measure would likely shrink the neutral regions, isolating how much of the suite's challenge comes from neutrality versus symmetry."],"forward_implications":["Algorithms compared on this suite are being tested on a landscape property—permutation invariance with neutrality—that the BBOB suite does not isolate, so rankings from this suite complement rather than repeat standard benchmark results.","The high internal rank correlation (around 0.8–0.9) means the 10 datasets behave like a coherent problem class, so a configuration chosen on one is likely to generalize to the others at the same dimension.","Because the number of distinct basins grows with $k$, higher-dimensional instances in the suite will separate algorithms by their ability to escape local minima, not just by convergence speed.","The stick-breaking transformation does not help default CMA-ES and sometimes hurts, implying that simply removing symmetry is not enough; the induced hierarchical dependence between variables becomes a new obstacle.","The K-Means++ baselines give a cheap external reference, letting black-box optimizers be compared directly against a specialized non-black-box clustering method on the same objective values."],"supporting_citations":[{"why":"Supplies the K-Means++ seeding algorithm whose 100 repeated runs provide the reference baseline values for every instance.","marker":"[1]"},{"why":"Supplies the best-practice criteria for benchmarking suites (diversity, representativeness, scalability, known solutions) that structure the paper's analysis.","marker":"[3]"},{"why":"Provides the modular CMA-ES implementation used to create the 128 configuration portfolio.","marker":"[9]"},{"why":"Provides the underlying experimenter and logging platform into which the proposed suite is integrated.","marker":"[10]"},{"why":"Establishes clustering problems as optimization benchmarks and supplies the earlier 27-instance set that this suite extends and standardizes.","marker":"[13]"},{"why":"Provides the BBOB problem suite used as the comparison baseline for both ranking correlations and landscape analysis.","marker":"[17]"},{"why":"Introduces exploratory landscape analysis, the feature-based method used to compare problem characteristics across suites.","marker":"[27]"},{"why":"Provides the implementation used to compute the exploratory landscape analysis features from sampled function values.","marker":"[32]"}],"fun_headline_variants":["Clustering benchmarks expose optimizer symmetry blind spots","New suite: neutral plateaus and symmetry for optimizer tests","Standardized clustering problems for fair optimizer comparison","Symmetry and neutrality: fresh challenges in benchmark suite","Clustering-based benchmarks reveal optimizer diversity gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that PCA-reducing every dataset to two dimensions and min-max normalizing it preserves the landscape features—permutation symmetry, neutrality, and multimodality—that make clustering problems interesting; if the projection distorts or destroys those features, the suite may no longer represent the class it claims to benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Clustering benchmarks expose optimizer symmetry blind spots","New suite: neutral plateaus and symmetry for optimizer tests","Standardized clustering problems for fair optimizer comparison","Symmetry and neutrality: fresh challenges in benchmark suite","Clustering-based benchmarks reveal optimizer diversity gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1486,"prompt_tokens":1082,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":698,"tokens_out":404,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:35:27.523764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the 10 datasets, compute the original high-dimensional clustering landscape, then compare its number of distinct local basins and its ELA feature vector to the PCA-reduced $[0,1]^2$ version used in the suite; if the reduced problem has fewer basins or a markedly different ELA profile, the standardization step is not neutral. A sharper test: construct a synthetic dataset with three well-separated high-dimensional clusters engineered so that PCA projects two clusters together, and check whether the projected problem still exhibits the same $k!$ symmetric structure and neutrality as the original.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the underlying experimenter and logging platform into which the proposed suite is integrated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the modular CMA-ES implementation used to create the 128 configuration portfolio."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes clustering problems as optimization benchmarks and supplies the earlier 27-instance set that this suite extends and standardizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BBOB problem suite used as the comparison baseline for both ranking correlations and landscape analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the implementation used to compute the exploratory landscape analysis features from sampled function values."}],"review_version":1}