Pith. sign in

REVIEW 2 major objections 2 minor 37 references

How to evaluate clustering with ground truth?

T0 review · 2 major / 2 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Centroid index is recommended as the preferred external measure for clustering evaluation when ground truth is available.

desk verdict Short review recommending centroid index for intuitiveness at cluster level, but with no comparisons or evidence to support the picks. read the letter →

arxiv 2606.27061 v1 pith:PAXIB36S submitted 2026-06-25 cs.AI

Pasi Fränti This is my paper · ORCID
classification cs.AI
keywords clusteringevaluationexternalvalidityindexescentroidindexgroundtruthset-matchingmeasuresclusteraccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reviews common external validity indexes for clustering when ground truth labels exist, with emphasis on set-matching-based measures. It concludes that the centroid index stands out because it operates at the cluster level and yields results that can be directly explained in terms of mismatched clusters. Readers should care since the choice of index shapes how researchers and practitioners judge the quality of clustering algorithms in real applications. When finer point-level detail is required, the pair-set index supplies a normalized score free of cluster-size bias, while clustering accuracy suits cases where every point must count equally.

What carries the argument

Centroid index (CI), a set-matching measure that counts how many clusters fail to align with the ground-truth partition at the cluster level.

What would settle it

A controlled user study in which domain experts consistently rate another index, such as the adjusted Rand index, as more useful or accurate than the centroid index across multiple datasets would challenge the recommendation.

Watch

Extended reading notes

Core claim

External indexes based on set matching evaluate how well a clustering matches known ground-truth partitions. The centroid index is the recommended choice because it is an intuitive cluster-level measure whose result can be explained directly. When a more detailed point-level score is needed, the pair-set index delivers a normalized value without bias from unequal cluster sizes, and clustering accuracy works if every data point must contribute equally to the score.

Load-bearing premise

That prioritizing intuitiveness and explainability at the cluster level, within the family of set-matching measures, is the right way to choose an evaluation index.

Editorial extensions

If this is right

  • Clustering results become directly interpretable in terms of which specific clusters mismatch the ground truth.
  • Normalized scores without size bias become available when the pair-set index is chosen instead.
  • Equal weighting of every data point is achieved by selecting clustering accuracy or similar set-matching measures.
  • Evaluation practice shifts away from indexes that lack cluster-level explainability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Widespread adoption of the centroid index could simplify comparisons across different clustering papers.
  • The emphasis on explainability at the cluster level may encourage similar design choices in other unsupervised evaluation settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript reviews common external validity indexes for clustering when ground truth is available, restricting attention to set-matching-based measures. It recommends the centroid index (CI) on grounds of cluster-level intuitiveness and explainability, proposes the pair-set index (PSI) when a normalized score unbiased by cluster size is required, and suggests clustering accuracy (ACC) or similar set-matching measures when every point should contribute equally.

Significance. A systematic review that supplies explicit, reproducible criteria for choosing among set-matching indexes could help standardize evaluation practice in unsupervised learning. The paper's explicit scoping to set-matching measures and its emphasis on cluster-level explainability are strengths if they are accompanied by concrete comparisons that demonstrate when CI outperforms alternatives on the stated criteria.

major comments (2)
  1. [Abstract] Abstract: the recommendations for CI, PSI, and ACC are stated without any comparative table, derivation, or empirical result; the central claim that CI is preferable therefore rests on unshown analysis.
  2. [Recommendation section] Recommendation section (or equivalent): the assertion that CI is 'intuitive' and 'explainable' is presented as decisive, yet no operational definition of these properties or head-to-head evaluation against other indexes (e.g., on bias, normalization, or sensitivity to cluster-size imbalance) is supplied.
minor comments (2)
  1. All indexes discussed should be accompanied by their original citations and, where possible, the exact formulas used in the review.
  2. A short paragraph justifying the exclusion of information-theoretic or pair-counting indexes outside the set-matching family would clarify the review's scope.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address the major comments point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the recommendations for CI, PSI, and ACC are stated without any comparative table, derivation, or empirical result; the central claim that CI is preferable therefore rests on unshown analysis.

    Authors: The abstract is intentionally concise. The manuscript reviews set-matching indexes and grounds the recommendations in their structural properties (cluster-level vs. point-level evaluation, normalization behavior, and cluster-size bias). To address the concern, we will add an explicit comparative table in the revised manuscript. revision: yes

  2. Referee: [Recommendation section] Recommendation section (or equivalent): the assertion that CI is 'intuitive' and 'explainable' is presented as decisive, yet no operational definition of these properties or head-to-head evaluation against other indexes (e.g., on bias, normalization, or sensitivity to cluster-size imbalance) is supplied.

    Authors: We employ 'intuitive' and 'explainable' to indicate that CI reports mismatches at the level of individual clusters rather than an aggregate score. We agree that operational definitions and direct comparisons would strengthen the presentation. In revision we will supply definitions of these terms and a head-to-head comparison on the listed criteria. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

The paper is a review of existing set-matching external validity indexes for clustering with ground truth. It presents no mathematical derivations, equations, fitted parameters, or predictions that could reduce to inputs by construction. The recommendation of centroid index (CI) rests on explicitly stated qualitative criteria (intuitiveness and explainability at cluster level), not on any self-referential fitting, self-citation chain, or ansatz. No load-bearing step matches any of the enumerated circularity patterns.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is a review paper that discusses existing indexes without introducing new parameters, axioms, or postulated entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to evaluate clustering with ground truth?." pith.science (2026). https://pith.science/paper/PAXIB36S

@misc{pith2026260627061,
  author       = {Pith},
  title        = {Pith review of: How to evaluate clustering with ground truth?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PAXIB36S}},
  note         = {Machine review of arXiv:2606.27061}
}
read the original abstract

External indexes can be used for cluster evaluation when ground truth is available. We review the most common external validity indexes focusing on set-matching-based measures. We recommend centroid index (CI), because it is an intuitive cluster-level measure with an explainable result. If we need a more fine-tuned, point-level measure, there are more choices. Pair-set index (PSI) provides a normalized score which is not biased by cluster sizes. If all points should matter equally, then clustering accuracy (ACC) or any other set-matching measure is suitable.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

  1. [1]

    Fränti and S

    P. Fränti and S. Sieranoja, Clustering accuracy, Applied Computing and Intelligence, 4 (1), 24 - 44, 2024

    2024

  2. [2]

    Kinnunen and H

    T. Kinnunen and H. Li, An overview of text-independent speaker recognition: From features to supervectors, Speech Communication, 52 (1), 12-40, 2010

    2010

  3. [3]

    Kinnunen, I

    T. Kinnunen, I. Sidoroff, M. Tuononen, and P. Fränti, Comparison of clustering methods: a case study of text -independent speaker modeling, Pattern Recognition Letters , 32 (13), 1604 -1617. October 2011

    2011

  4. [4]

    Fränti, C

    P. Fränti, C. Cariou and Q. Zhao, Cluster overlap as objective function, CMC-Computers, Materials & Continua, 66534, 1-28, 2025

    2025

  5. [5]

    Fränti, M

    P. Fränti, M. Rezaei and Q. Zhao, Centroid index: cluster level similarity measure, Pattern Recognition, 47 (9), 3034-3045, 2014

    2014

  6. [6]

    Fränti, Efficiency of random swap clustering, Journal of Big Data, 5:13, 1-29, 2018

    P. Fränti, Efficiency of random swap clustering, Journal of Big Data, 5:13, 1-29, 2018

    2018

  7. [7]

    Fränti, Genetic algorithm with deterministic crossover for vector quantization, Pattern Recognition Letters, 21 (1), 61-68, 2000

    P. Fränti, Genetic algorithm with deterministic crossover for vector quantization, Pattern Recognition Letters, 21 (1), 61-68, 2000

    2000

  8. [8]

    Likas, N

    A. Likas, N. Vlassis, J.J. Verbeek, The global k-means clustering algorithm, Pattern Recognition, 36 (2), 451-461, 2003

    2003

Show all 37 references
  1. [9]

    Fränti and S

    P. Fränti and S. Sieranoja, How much k-means can be improved by using better initialization and repeats?, Pattern Recognition, 93, 95-112, 2019

    2019

  2. [10]

    Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J

    P. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J. Comput. Appl. Math. 20, 53–65, 1987

    1987

  3. [11]

    Calinski, J

    T. Calinski, J. Harabasz, A dendrite method for cluster analysis, Commun. Stat. 3, 1–27, 1974

    1974

  4. [12]

    Zhao and P

    Q. Zhao and P. Fränti, WB -index: a sum -of-squares based index for cluster validity, Data & Knowledge Engineering, 92, 77-89, July 2014

    2014

  5. [13]

    Hämäläinen, S

    J. Hämäläinen, S . Jauhiainen, T . Kärkkäinen, Comparison of internal clustering validation indices for prototype-based clustering, Algorithms 10 (3), 105, 2017

    2017

  6. [14]

    Davies, D

    D. Davies, D. Bouldin, Cluster separation measure, IEEE Trans. Pattern Anal ysis and Machine Intelligence, 1 (2), 95–104, 1979

    1979

  7. [15]

    Niemelä, S

    M. Niemelä, S. Äyrämö, and T. Kärkkäinen, Toolbox for distance estimation and cluster validation on data with missing values. IEEE Access, 10, pp.352-367, 2021

    2021

  8. [16]

    Bagirov, R.M

    A.M. Bagirov, R.M. Aliguliyev, N. Sultanova, Finding compact and well -separated clusters: Clustering using silhouette coefficients, Pattern Recognition, 135, 109144, March 2023

    2023

  9. [17]

    Ikotun, F

    A.M. Ikotun, F. Habyarimana, A.E. Ezugwu, Cluster validity indices for automatic clustering: A comprehensive review, Heliyon, 11 (2), e41953, January 2025

    2025

  10. [18]

    K -means properties on six clustering benchmark datasets

    P. Fränti and S. Sieranoja, "K -means properties on six clustering benchmark datasets", Applied Intelligence, 48 (12), 4743-4759, December 2018

    2018

  11. [19]

    Hubert and P

    L. Hubert and P. Arabie, Comparing partitions. Journal of Classification, 2 (1), 193-218 (1985)

    1985

  12. [20]

    Kvålseth, Entropy and correlation: some comments, IEEE Trans

    T.O. Kvålseth, Entropy and correlation: some comments, IEEE Trans. Syst ems, Man and Cybernetic, 17 (3), 517-519, 1987

    1987

  13. [21]

    Rand, Objective criteria for the evaluation of clustering methods

    W.M. Rand, Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association 66: 846–850, 1971

    1971

  14. [22]

    Warrens and H

    M.J. Warrens and H. van der Hoef, Understanding the Rand index. Advanced Studies in Classification and Data Science, pp. 301-313, Springer, Singapore, 2020

    2020

  15. [23]

    Warrens and H

    M.J. Warrens and H. van der Hoef, Understanding the adjusted Rand index and other partition comparison indices based on counting object pairs. Journal of Classification, 39, 487–509, 2022

    2022

  16. [24]

    Rezaei and P

    M. Rezaei and P. Fränti, Set matching measures for external cluster validity, IEEE Trans. on Knowledge and Data Engineering, 28 (8), 2173-2186, August 2016

    2016

  17. [25]

    van Dongen, Performance criteria for graph clustering and Markov cluster experiments, Technical Report INSR0012, Centrum voor Wiskunde en Informatica, 2000

    S. van Dongen, Performance criteria for graph clustering and Markov cluster experiments, Technical Report INSR0012, Centrum voor Wiskunde en Informatica, 2000

    2000

  18. [26]

    Meila and D

    M. Meila and D. Heckerman, An experimental comparison of model based clustering methods, Machine Learning, 41(1-2), pp. 9–29, 2001

    2001

  19. [27]

    D. Cai, X. He, J. Han, Document clustering using locality preserving indexing, IEEE Trans. Knowledge and Data Engineering, 17 (12), 2005, 1624-1637

    2005

  20. [28]

    Rendón, I

    E. Rendón, I. Abundez, A. Arizmendi, and E.M. Quiroz, Internal versus external cluster validation indexes. Int. Journal of computers and communications, 5 (1), 27-34, 2011

    2011

  21. [29]

    de Souto, A.L.V

    M.C.P. de Souto, A.L.V. Coelho, K. Faceli, T.C. Sakata, V. Bonadia, and I.G. Costa, A comparison of external clustering evaluation indices in the context of imbalanced data sets, 2012 Brazilian Symposium on Neural Networks, pp. 49-54, 2012

    2012

  22. [30]

    Zhao and P

    Q. Zhao and P. Fränti, Centroid ratio for pairwise random swap clustering algorithm, IEEE Trans. on Knowledge and Data Engineering, 26 (5), 1090-1101, May 2014

    2014

  23. [31]

    Fränti and M

    P. Fränti and M. Rezaei, Generalized centroid index to different clustering models, S+SSPR’16, Merida, Mexico, LNCS 10029, 285-296, November 2016

    2016

  24. [32]

    Slonim and N

    N. Slonim and N. Tishby, Document clustering using word clusters via the information bottleneck method. Int. ACM SIGIR Conf. on Research and development in information retrieval, 208-215, 2000

    2000

  25. [33]

    Kuhn, The Hungarian method for the assignment problem, Naval Res

    H.W. Kuhn, The Hungarian method for the assignment problem, Naval Res. Logist. Quart., 2 (1- 2), 83–97, 1955

    1955

  26. [34]

    Munkres, Algorithms for the assignment and transportation problems, J

    J. Munkres, Algorithms for the assignment and transportation problems, J. Soc. Ind. Appl. Math., 5 (1), 32–38, 1957

    1957

  27. [35]

    Jonker and A

    R. Jonker and A. Volgenant, A shortest augmenting path algorithm for dense and sparse linear assignment problems, Computing, 38 (4): 325–340, December 1987. doi:10.1007/BF02278710. S2CID 7806079

    1987 doi

  28. [36]

    Mahmoudi and D

    A. Mahmoudi and D. Jemielniak, Proof of biased behavior of normalized mutual information, Scientific Reports, 14:9021, 1-17, 2024

    2024

  29. [37]

    Rezaei and P

    M. Rezaei and P. Fränti, Can the number of clusters be determined by external indices?, IEEE Access, 8 (1), 89239-89257, December 2020

    2020

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.