REVIEW 2 major objections 6 minor 2 references
A Critical Note on the Evaluation of Clustering Algorithms
T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Using classification labels as ground truth for clustering can reward wrong partitions and mislead algorithm comparisons, the paper demonstrates on synthetic and real data.
desk verdict A clear, reproducible cautionary note that class labels are not cluster labels, but it overreaches on a t-SNE claim and the central point is not new. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the contrast between internal and external validation criteria. Internal criteria (Davies-Bouldin index, silhouette coefficient) judge a partition purely by geometric compactness and separation; external criteria (ARI, MI/NMI, AMI) judge it by agreement with a supplied 'ground truth' partition. The experiments hold the data fixed and vary only the partition, so disagreement between the two families of scores isolates the effect of using class labels as ground truth. A second mechanism is a geometric probability model: represent each class as a circle of radius $r$ in a $w \times w$ box; the chance that $k$ such classes are all pairwise disjoint is approximately $e^{-2\pi r^2 k(k-1)/w^2}$, which quantifies how quickly class labels cease to be a usable proxy for cluster structure.
What would settle it
Audit a broad collection of public classification datasets: for each, compute the agreement (e.g., ARI) between the class-label partition and the partition that optimizes an internal criterion such as silhouette width, and check whether external and internal metrics rank clustering algorithms consistently across datasets. If most datasets show high agreement and consistent rankings, the paper's warning would lose much of its force.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a 'chain of errors' in standard clustering evaluation: class labels are not defined by the data distribution's natural group structure, yet they are fed into external metrics as if they were true cluster labels, and algorithms that score well under those metrics are then declared superior even when their partitions contradict the visible density structure. The synthetic dataset SD_2 makes this concrete: it has two class labels but three well-separated Gaussian clusters, and K-means with $k=2$ or DBSCAN with a looser neighborhood scores 1.0 on ARI, NMI, and AMI while splitting a dense cluster in half, whereas the visually correct three-cluster solution scores lower. The real-world cases show the same pattern in both directions: overlapping classes that should form one cluster receive near-zero label-based scores, and a single class split into several dense regions is penalized when an algorithm correctly separates those regions. A formal calculation with $k$ circular classes in a square gives the probability that all classes are mutually disjoint as approximately $e^{-2\pi r^2 k(k-1)/w^2}$, so label-cluster mismatch becomes the rule as the number of classes grows.
Load-bearing premise
The demonstrations assume that the clusters visible in two-dimensional plots, or in t-SNE/PCA projections, are the true cluster structure that class labels should match; if that visual judgment is wrong, the metric contradictions lose their reference point.
Editorial extensions
If this is right
- Benchmark comparisons that use classification labels as ground truth may rank algorithms by how well they reproduce class boundaries rather than by how well they find natural groups.
- High values of ARI, NMI, or AMI should not be taken as evidence of good clustering unless the label partition itself has been checked against the data's density structure.
- Tuning clustering hyperparameters to maximize label-based scores can actively select worse partitions, as the SD_2 experiment shows.
- Researchers should inspect benchmark datasets before use and prefer purpose-built synthetic datasets with controlled, known cluster structure.
- Reported performance differences between clustering algorithms on classification datasets need to be re-audited before being trusted.
Reading between the lines
- This critique extends beyond the specific metrics tested: any workflow that tunes or selects a clustering model against classification labels inherits the same bias, so published performance gaps may partly measure label alignment rather than clustering ability.
- The same label-as-ground-truth problem is likely present in other unsupervised evaluations that borrow classification labels, such as representation learning, anomaly detection, or community detection benchmarks.
- The probability calculation implies the problem worsens with the number of classes, so high-cardinality classification datasets are the riskiest choices for clustering benchmarks.
- A direct extension would be to generate synthetic benchmarks with known cluster structure and varying class overlap, then measure how far external-metric rankings drift from internal-metric rankings as overlap increases; this could produce a practical threshold for when class labels are unusable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues against a widespread evaluation practice in clustering research: using class labels from classification datasets as the ground truth for external cluster validity metrics. It defines the metrics used (DBI, SC, ARI, MI/NMI/AMI), then presents a synthetic 2D dataset in which one class is generated from two well-separated Gaussian components, and two real-world datasets (weight-height, accelerometer) together with a PCA/t-SNE analysis of the Vertebral Column dataset, to show that class labels can disagree with the apparent cluster structure. It also gives a simple probabilistic model in which classes are circles in a bounded box and derives that the probability of all classes being disjoint decreases rapidly with the number of classes. The paper concludes that benchmark datasets should be revisited and that controlled synthetic data should be preferred for principled clustering evaluation.
Significance. If the central claim is accepted, the paper is a useful warning: external metrics such as ARI/NMI/AMI are only meaningful when the reference partition actually corresponds to the clustering objective. The synthetic SD_2 example is a clean, well-controlled counterexample: the generative construction fixes three Gaussian components, so the disagreement between class labels and cluster structure is not a matter of taste. The paper is also commendably transparent in reporting parameter settings (k-dist for DBSCAN, parameter ranges in Table 1) and in stating the assumptions of the probability model. The main limitation is that the real-data case studies rely heavily on visual judgments of 2D plots, and in the Vertebral Column example the inference from low-dimensional projections to the original space is not justified. The contribution is a critical note rather than a new algorithm or a comprehensive benchmark study, but if the overstatements are corrected the paper can serve its intended purpose.
major comments (2)
- [Real-World Datasets: Splitting Data] The claim that 'in the original space, it is sure that objects in class abnormal are not distributed in the form of a single cluster' (text following Figure 5) is not supported by the presented evidence. t-SNE can produce separated blobs even for data without cluster structure, and the PCA view uses only two of six principal components, so it cannot certify disconnectedness in the remaining dimensions. Since this inference is used to conclude that the Vertebral Column class labels are unsuitable for clustering evaluation, the paper should either replace it with a direct analysis of the original 6D data (e.g., a multi-modality or clusterability test, or density-based clustering on the original features) or explicitly weaken the claim to a statement about the projected views. The synthetic SD_2 example remains a valid counterexample to using class labels, but the real-data part of the paper currently overstates its certainty.
- [Real-World Datasets: Overlapping Data and Splitting Data (Figures 3 and 4)] The real-world demonstrations define the 'reasonable' clustering by visual inspection of 2D scatter plots. In the Weight-Height case, the assertion that the data is 'more like a single Gaussian distribution' is not backed by any statistical test; in the Accelerometer case, the claim that descend_stairs splits into two dense regions and that one region should merge with climb_stairs is likewise purely visual. Because the paper's argument is precisely that class labels are the wrong reference, it needs an explicit, operational criterion for what counts as the 'natural' cluster structure (e.g., connected components of a density level set, or an internal validity index) in these examples. Without such a criterion, the real-data demonstrations only show that external metrics respond to the class labels, which is definitional rather than evidence of misleading evaluation.
minor comments (6)
- [Introduction] The phrase 'without any sign of decease' should be 'without any sign of ceasing' (or 'abating').
- [Real-World Datasets: Splitting Data] The section title contains 'real word' where 'real world' is intended.
- [Table 2] In the K-means (k=1) row, DBI is reported as -2 and SC is blank; since internal criteria are undefined for a single cluster, the table should use a placeholder such as 'N/A' and explain it in the caption, consistently with the footnote.
- [Discussion, Equations (8)-(11)] The independence assumption between pairs of centers is not true even approximately for a fixed packing, and the boundary correction is only mentioned in passing. The paper should note that Eq. (9) is a heuristic approximation, not a rigorous bound, although the simulation in Figure 6 appears consistent with it.
- [Figure 5] The t-SNE analysis does not report hyperparameters such as perplexity, number of iterations, or initialization, and the PCA view does not report the component loadings; this limits reproducibility of the visualization claim.
- [References] The reference for Zhang and Yuan (2018) has an obvious page-number formatting error ('2586125873' should presumably be '25861-25873').
Circularity Check
No significant circularity: the central demonstrations use reference partitions independent of the class labels, and the only self-citation is not load-bearing.
full rationale
The paper's central claim is that class labels from classification tasks need not coincide with cluster structure, and that external metrics scored against those labels can therefore disagree with visually or density-based cluster structure. This claim is not derived from itself. For the synthetic SD_2 dataset, the reference partition is fixed by the generative construction: Class 2 is built from two Gaussian distributions, giving three natural components, and the paper shows that external metrics evaluated against the two class labels favor the wrong k=2 solution. For the real-world cases, the reference partition is established by visual or density inspection of the data (e.g., 'it is clear that there is some overlapping region between the two classes, making the distribution of the data more like a single Gaussian distribution'), which is independent of the class labels. The observed disagreement between external metric scores and those reference partitions is a contingent empirical finding, not a logical reduction to the paper's premises. The probability analysis in Equations (8)-(11) is a self-contained combinatorial approximation. The main weakness is methodological rather than circular: the Vertebral Column case infers structure in the original 6D space from t-SNE and a purpose-chosen PCA projection, and the real-data demonstrations rely on accepting visual judgments as ground truth. That is a limitation, but it is not a circularity because the visual reference is not the same object as the conclusion being established. The only self-citation, 'the challenge of clustering analysis mainly comes from factors such as irregular clusters, strong noise and high dimensionality (Zhang and Yuan 2018),' appears in the introduction and is not load-bearing for the evaluation critique; it merely motivates the difficulty of clustering. Therefore no circular step is exhibited, and the paper's central argument has independent content.
Assumptions & free parameters
free parameters (8)
- SD_2 class sizes =
~800, 1000, 1200 objects
- SD_2 Gaussian standard deviations =
[0.008,0.002], [0.007,0.003], [0.007,0.004]
- K-means k =
2 or 3
- DBSCAN Eps and minPts =
e.g., Eps=0.025, minPts=4
- Weight-height sample size =
400 samples
- Accelerometer activity selection =
climb_stairs and descend_stairs
- Probability model circle radius r and space width w =
r=0.01, w=1.0
- Simulation trials =
10,000 per k
assumptions (6)
- domain assumption The events that two randomly placed circles do not overlap are independent.
- standard math For r << w, the first-order Taylor expansion e^x ≈ 1 + x is valid.
- domain assumption Visual inspection of 2D plots reliably reveals the number of clusters and cluster membership.
- domain assumption t-SNE and PCA projections preserve the cluster structure relevant to the original high-dimensional space.
- ad hoc to paper Classes can be modeled as circles of identical radius for the probability analysis.
- domain assumption The standard implementations and definitions of K-means, DBSCAN, and the evaluation metrics are correct.
Cite this review
Pith. "Pith review of A Critical Note on the Evaluation of Clustering Algorithms." pith.science (2026). https://pith.science/paper/LWCG6N4X
@misc{pith2026190803782,
author = {Pith},
title = {Pith review of: A Critical Note on the Evaluation of Clustering Algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/LWCG6N4X}},
note = {Machine review of arXiv:1908.03782}
}
read the original abstract
Experimental evaluation is a major research methodology for investigating clustering algorithms and many other machine learning algorithms. For this purpose, a number of benchmark datasets have been widely used in the literature and their quality plays a key role on the value of the research work. However, in most of the existing studies, little attention has been paid to the properties of the datasets and they are often regarded as black-box problems. For example, it is common to use datasets intended for classification in clustering research and assume class la-bels as the ground truth for judging the quality of cluster-ing. In our work, with the help of advanced visualization and dimension reduction techniques, we show that this practice may seriously compromise the research quality and produce misleading results. We suggest that the applicability of existing benchmark datasets should be carefully revisited and significant efforts need to be devoted to improving the current practice of experimental evaluation of clustering algorithms to ensure an essential match between algorithms and problems.
Figures
Reference graph
Works this paper leans on
-
[2009]
Knowledge and Information Systems 19(3): 361–394
Characteriza- tion and Evaluation of Similarity M easures for Pairs of Cluster- ings. Knowledge and Information Systems 19(3): 361–394. Pearson, K. 1901 . LIII. On Lines and Planes of Closest F it to Systems of Points in S pace. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2(11): 559–572. Rousseeuw, P. J
work page 1901
-
[2019]
In Wikipedia, The Free Encyclopedia
Cluster Analysis. In Wikipedia, The Free Encyclopedia . Retrieved August 26, 2019, from https://en.wikipedia.org/w/index.php?title=Cluster_analysis&oldi d=909008543. Wiwie, C., Baumbach, J., and Röttger, R
work page 2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.