{"id":"184ddc54-8b87-48d1-bc5b-021001efaebd","arxiv_id":"1908.03782","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":8,"one_line_summary":"Class labels from classification datasets are often inconsistent with actual cluster structure, so using them as ground truth in clustering evaluation can mislead.","lead":"This paper argues that using class labels from classification datasets as ground truth for evaluating clustering algorithms can produce misleading results. It shows with examples that labels often do not match natural cluster structure, and recommends revisiting common benchmark practices.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The demonstrations that class labels are misleading rely on visually inferred ground truth, and the t-SNE/PCA case in 'Real-World Datasets: Splitting Data' overstates what can be concluded about the original space; this is a real limitation, but the synthetic counterexample keeps the central claim…","rationale":"The reader's weakest-assumption analysis correctly identifies that the case studies depend on visual judgments of cluster structure. My stress test agrees with that assessment and sharpens it: the Vertebral Column paragraph explicitly moves from a t-SNE/PCA visualization to a claim about the original high-dimensional space, which is not warranted by the evidence shown. That is a genuine limitation and supports the conditional verdict. However, it does not overturn the paper's central point. The synthetic dataset SD_2 is constructed from three well-separated Gaussian components, with one class label deliberately spanning two components; in that setting the 'natural' clustering is known from the generating process rather than from visual inspection alone, and the metric reversal (external criteria favoring k=2 over k=3) is a valid counterexample to the unguarded use of class labels as clustering ground truth. The paper also explicitly frames its claim with 'may' rather than 'always', and its definition of a cluster as a densely connected group is stated clearly at the outset. Thus the main conceptual argument survives the identified weakness. The remaining concern is scope and evidentiary overreach in the high-dimensional demonstration, not a fatal flaw in the central claim. No change to the reader's conditional verdict is needed.","tokens_in":10632,"tokens_out":4279,"duration_ms":54463,"concrete_test":"On the Vertebral Column dataset, replace visual inspection with an explicit, parameter-defined cluster notion in the original 6-dimensional space: run HDBSCAN over a grid of min_cluster_size values (e.g., 5, 10, 20, 40) using the Euclidean metric, and record how the 'abnormal' class is partitioned. If for no reasonable parameterization does 'abnormal' split into two or more clusters, then the t-SNE/PCA claim of a split class is unsupported and the case study does not demonstrate misleading external metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference is that class labels misrepresent the natural cluster structure of a dataset, and this inference needs a reference partition that is not itself defined by the labels. In the synthetic SD_2 case the reference is secure because the generative construction fixes three Gaussian components. In the real-data cases, however, the reference is established visually. The weakest step is in the Vertebral Column analysis: from a t-SNE plot and a purpose-chosen PCA projection of the 6-dimensional data, the paper asserts that 'it is sure that objects in class abnormal are not distributed in the form of a single cluster' in the original space. t-SNE is known to create separated blobs even for high-dimensional data without cluster structure, and a two-component PCA projection cannot certify disconnectedness in the remaining dimensions. If that inference is wrong, the claim that external metrics are misleading for real-world classification datasets loses its reference point; the paper is then only showing that external metrics score agreement with labels, which is definitional. The same reliance on visual judgment appears in the Weight-Height and Accelerometer examples. Because clustering has no objective ground truth without specifying a clustering criterion, the force of 'may seriously compromise research quality' depends on accepting the visual or density-based notion of clusters as the correct one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues against a widespread evaluation practice in clustering research: using class labels from classification datasets as the ground truth for external cluster validity metrics. It defines the metrics used (DBI, SC, ARI, MI/NMI/AMI), then presents a synthetic 2D dataset in which one class is generated from two well-separated Gaussian components, and two real-world datasets (weight-height, accelerometer) together with a PCA/t-SNE analysis of the Vertebral Column dataset, to show that class labels can disagree with the apparent cluster structure. It also gives a simple probabilistic model in which classes are circles in a bounded box and derives that the probability of all classes being disjoint decreases rapidly with the number of classes. The paper concludes that benchmark datasets should be revisited and that controlled synthetic data should be preferred for principled clustering evaluation.","tokens_in":10953,"tokens_out":7242,"duration_ms":75408,"significance":"If the central claim is accepted, the paper is a useful warning: external metrics such as ARI/NMI/AMI are only meaningful when the reference partition actually corresponds to the clustering objective. The synthetic SD_2 example is a clean, well-controlled counterexample: the generative construction fixes three Gaussian components, so the disagreement between class labels and cluster structure is not a matter of taste. The paper is also commendably transparent in reporting parameter settings (k-dist for DBSCAN, parameter ranges in Table 1) and in stating the assumptions of the probability model. The main limitation is that the real-data case studies rely heavily on visual judgments of 2D plots, and in the Vertebral Column example the inference from low-dimensional projections to the original space is not justified. The contribution is a critical note rather than a new algorithm or a comprehensive benchmark study, but if the overstatements are corrected the paper can serve its intended purpose.","major_comments":[{"comment":"The claim that 'in the original space, it is sure that objects in class abnormal are not distributed in the form of a single cluster' (text following Figure 5) is not supported by the presented evidence. t-SNE can produce separated blobs even for data without cluster structure, and the PCA view uses only two of six principal components, so it cannot certify disconnectedness in the remaining dimensions. Since this inference is used to conclude that the Vertebral Column class labels are unsuitable for clustering evaluation, the paper should either replace it with a direct analysis of the original 6D data (e.g., a multi-modality or clusterability test, or density-based clustering on the original features) or explicitly weaken the claim to a statement about the projected views. The synthetic SD_2 example remains a valid counterexample to using class labels, but the real-data part of the paper currently overstates its certainty.","section":"Real-World Datasets: Splitting Data"},{"comment":"The real-world demonstrations define the 'reasonable' clustering by visual inspection of 2D scatter plots. In the Weight-Height case, the assertion that the data is 'more like a single Gaussian distribution' is not backed by any statistical test; in the Accelerometer case, the claim that descend_stairs splits into two dense regions and that one region should merge with climb_stairs is likewise purely visual. Because the paper's argument is precisely that class labels are the wrong reference, it needs an explicit, operational criterion for what counts as the 'natural' cluster structure (e.g., connected components of a density level set, or an internal validity index) in these examples. Without such a criterion, the real-data demonstrations only show that external metrics respond to the class labels, which is definitional rather than evidence of misleading evaluation.","section":"Real-World Datasets: Overlapping Data and Splitting Data (Figures 3 and 4)"}],"minor_comments":[{"comment":"The phrase 'without any sign of decease' should be 'without any sign of ceasing' (or 'abating').","section":"Introduction"},{"comment":"The section title contains 'real word' where 'real world' is intended.","section":"Real-World Datasets: Splitting Data"},{"comment":"In the K-means (k=1) row, DBI is reported as -2 and SC is blank; since internal criteria are undefined for a single cluster, the table should use a placeholder such as 'N/A' and explain it in the caption, consistently with the footnote.","section":"Table 2"},{"comment":"The independence assumption between pairs of centers is not true even approximately for a fixed packing, and the boundary correction is only mentioned in passing. The paper should note that Eq. (9) is a heuristic approximation, not a rigorous bound, although the simulation in Figure 6 appears consistent with it.","section":"Discussion, Equations (8)-(11)"},{"comment":"The t-SNE analysis does not report hyperparameters such as perplexity, number of iterations, or initialization, and the PCA view does not report the component loadings; this limits reproducibility of the visualization claim.","section":"Figure 5"},{"comment":"The reference for Zhang and Yuan (2018) has an obvious page-number formatting error ('2586125873' should presumably be '25861-25873').","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short critical note rather than a full research contribution. Its central point is not entirely new, but the synthetic counterexample is clean and the paper is transparent about parameter choices. The main risk is the overstatement in the Vertebral Column section; after adding quantitative checks in the original feature space and toning down the language, it could be suitable for publication as a position or critical note. I would not recommend rejection solely for lack of novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Zhang, Zhong, Yuan note on clustering evaluation. Short version: it’s a useful cautionary tale, but not the exposé the authors seem to think. The core point—class labels are not cluster labels—is well known in the clustering community, and the paper doesn’t cite the prior critiques that make this point. What it adds is concrete, reproducible examples, and one of them is genuinely neat.\n\nThe synthetic SD_2 experiment is the strongest part. Three Gaussian components are generated; the data are labeled as two classes. K-means with k=2 (or DBSCAN with the wrong eps) scores perfect ARI/NMI against the class labels, while the correct k=3 partition scores much lower. That’s a clean illustration of how an external metric can reward a structurally wrong result when the reference partition is defined by class labels. The subsequent real-data examples—weight-height overlap, accelerometer split-class, and the vertebral column case—extend the point, but they depend on the reader accepting visually determined cluster structure as ground truth. Clustering is ill-posed without a specific criterion, so that’s a condition, not a given.\n\nThe softest spot is the vertebral column analysis. From a t-SNE plot and a selected two-component PCA projection, the authors assert it is “sure” that class abnormal is not a single cluster in the original 6D space. That’s an overreach. t-SNE is known to manufacture separated blobs even for data without cluster structure, and a 2D view cannot certify disconnectedness in the remaining dimensions. If that example is discounted, the real-data part weakens to “external metrics score agreement with labels,” which is definitional. The synthetic example still stands, so the central claim survives, but only with the force of a warning, not a proof.\n\nThe probability model in the Discussion is a rough back-of-the-envelope calculation. It’s correct under the stated circle model, and the simulations match. That’s fine, but it’s not a general result—equal radii, independent overlaps, and boundary effects are swept under the rug. The paper itself acknowledges the approximation, which is good.\n\nHow does this all balance? The paper is honest and clearly written. It would be a reasonable workshop paper or a brief correspondence, and it deserves a serious referee—though the referee should ask for (a) the t-SNE inference to be softened, (b) citations to earlier work on this exact issue, and (c) a statement that the real-data conclusions are conditional on a chosen clustering criterion. I wouldn’t fight for it in a top venue, but it’s not a desk reject.\n\nRecommendation: send it out, with the above revision requests.","headline":"A clear, reproducible cautionary note that class labels are not cluster labels, but it overreaches on a t-SNE claim and the central point is not new.","tokens_in":11416,"tokens_out":3532,"would_cite":true,"duration_ms":36994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using classification labels as ground truth for clustering can reward wrong partitions and mislead algorithm comparisons, the paper demonstrates on synthetic and real data.","keywords":["clustering evaluation","class labels","ground truth","external validation metrics","internal clustering criteria","benchmark datasets","unsupervised learning","adjusted rand index"],"falsifier":"Audit a broad collection of public classification datasets: for each, compute the agreement (e.g., ARI) between the class-label partition and the partition that optimizes an internal criterion such as silhouette width, and check whether external and internal metrics rank clustering algorithms consistently across datasets. If most datasets show high agreement and consistent rankings, the paper's warning would lose much of its force.","tokens_in":10451,"feed_emoji":"📊","tokens_out":13233,"duration_ms":120412,"temperature":0.7,"pith_summary":"The paper argues that a widespread evaluation practice in clustering research is unsound: taking the class labels of classification datasets as the ground truth for judging cluster quality. A class label records a property of each object, while clustering is about the density structure of the whole data distribution, and the two need not coincide. Through a synthetic dataset and three real-world datasets, the authors show that label-based scores such as the adjusted Rand index, normalized mutual information, and adjusted mutual information can rank a visually worse clustering above a visually better one, while internal geometric scores such as the Davies-Bouldin index and silhouette coefficient point the other way. A simple probability calculation with circular class regions reinforces the point: as the number of classes grows, class regions are very likely to overlap, so class labels will rarely match true clusters. The practical stakes are that published comparisons of clustering algorithms on classification benchmarks may be rewarding label reproduction rather than cluster discovery.","feed_headline":"Class labels mislead clustering benchmarks, case studies show","feed_subtitle":"On three datasets, label-based scores reward worse partitions while internal geometric scores point the other way.","key_machinery":"The load-bearing mechanism is the contrast between internal and external validation criteria. Internal criteria (Davies-Bouldin index, silhouette coefficient) judge a partition purely by geometric compactness and separation; external criteria (ARI, MI/NMI, AMI) judge it by agreement with a supplied 'ground truth' partition. The experiments hold the data fixed and vary only the partition, so disagreement between the two families of scores isolates the effect of using class labels as ground truth. A second mechanism is a geometric probability model: represent each class as a circle of radius $r$ in a $w \\times w$ box; the chance that $k$ such classes are all pairwise disjoint is approximately $e^{-2\\pi r^2 k(k-1)/w^2}$, which quantifies how quickly class labels cease to be a usable proxy for cluster structure.","core_discovery":"On the paper's own terms, the central discovery is a 'chain of errors' in standard clustering evaluation: class labels are not defined by the data distribution's natural group structure, yet they are fed into external metrics as if they were true cluster labels, and algorithms that score well under those metrics are then declared superior even when their partitions contradict the visible density structure. The synthetic dataset SD_2 makes this concrete: it has two class labels but three well-separated Gaussian clusters, and K-means with $k=2$ or DBSCAN with a looser neighborhood scores 1.0 on ARI, NMI, and AMI while splitting a dense cluster in half, whereas the visually correct three-cluster solution scores lower. The real-world cases show the same pattern in both directions: overlapping classes that should form one cluster receive near-zero label-based scores, and a single class split into several dense regions is penalized when an algorithm correctly separates those regions. A formal calculation with $k$ circular classes in a square gives the probability that all classes are mutually disjoint as approximately $e^{-2\\pi r^2 k(k-1)/w^2}$, so label-cluster mismatch becomes the rule as the number of classes grows.","pith_inferences":["This critique extends beyond the specific metrics tested: any workflow that tunes or selects a clustering model against classification labels inherits the same bias, so published performance gaps may partly measure label alignment rather than clustering ability.","The same label-as-ground-truth problem is likely present in other unsupervised evaluations that borrow classification labels, such as representation learning, anomaly detection, or community detection benchmarks.","The probability calculation implies the problem worsens with the number of classes, so high-cardinality classification datasets are the riskiest choices for clustering benchmarks.","A direct extension would be to generate synthetic benchmarks with known cluster structure and varying class overlap, then measure how far external-metric rankings drift from internal-metric rankings as overlap increases; this could produce a practical threshold for when class labels are unusable."],"forward_implications":["Benchmark comparisons that use classification labels as ground truth may rank algorithms by how well they reproduce class boundaries rather than by how well they find natural groups.","High values of ARI, NMI, or AMI should not be taken as evidence of good clustering unless the label partition itself has been checked against the data's density structure.","Tuning clustering hyperparameters to maximize label-based scores can actively select worse partitions, as the SD_2 experiment shows.","Researchers should inspect benchmark datasets before use and prefer purpose-built synthetic datasets with controlled, known cluster structure.","Reported performance differences between clustering algorithms on classification datasets need to be re-audited before being trusted."],"supporting_citations":[{"why":"Source of the classification datasets used in the case studies.","marker":"Dua and Graff 2019"},{"why":"Supplies the K-means algorithm used in all three case studies.","marker":"Hartigan and Wong 1979"},{"why":"Supplies DBSCAN, the density-based clustering algorithm, and the density-connected notion of a cluster used throughout.","marker":"Ester et al. 1996"},{"why":"Previous study cited for the practice of sweeping parameters to maximize agreement with class labels, the practice shown to select worse clusters.","marker":"Maurus and Plant 2016"},{"why":"Internal metric that scores a partition by compactness and separation, used as the geometric reference.","marker":"Davies and Bouldin 1979"},{"why":"Internal silhouette metric used as the second geometric reference.","marker":"Rousseeuw 1987"},{"why":"External ARI metric that rewards agreement with class labels, one of the scores shown to mislead.","marker":"Hubert and Arabie 1985"},{"why":"External NMI metric used to score agreement with class labels.","marker":"Strehl and Ghosh 2002"},{"why":"External AMI metric, chance-corrected, used to score agreement with class labels.","marker":"Vinh, Epps and Bailey 2009"},{"why":"t-SNE visualization used to reveal that the 'abnormal' class splits into multiple dense regions.","marker":"Maaten and Hinton 2008"}],"fun_headline_variants":["Label-cluster mismatch skews clustering algorithm evaluation","Two class labels, three real clusters: metrics reward wrong split","Benchmark datasets may mislead clustering studies, analysis finds","When class labels don't match clusters, evaluation scores mislead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstrations assume that the clusters visible in two-dimensional plots, or in t-SNE/PCA projections, are the true cluster structure that class labels should match; if that visual judgment is wrong, the metric contradictions lose their reference point.","fun_headline_variants_meta":{"raw":{"variants":["Label-cluster mismatch skews clustering algorithm evaluation","Two class labels, three real clusters: metrics reward wrong split","Benchmark datasets may mislead clustering studies, analysis finds","When class labels don't match clusters, evaluation scores mislead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000433,"raw_usage":{"total_tokens":2204,"prompt_tokens":938,"completion_tokens":1266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1211}},"tokens_in":554,"tokens_out":1266,"duration_ms":13733,"temperature":1.0,"reasoning_tokens":1211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:01:23.282569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a broad collection of public classification datasets: for each, compute the agreement (e.g., ARI) between the class-label partition and the partition that optimizes an internal criterion such as silhouette width, and check whether external and internal metrics rank clustering algorithms consistently across datasets. If most datasets show high agreement and consistent rankings, the paper's warning would lose much of its force.","supporting_citations":[],"review_version":1}