REVIEW 5 major objections 6 minor 36 references
Simultaneous Estimation of Number of Clusters and Feature Sparsity in Clustering High-Dimensional Data
T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that a single resampling score, S4, can simultaneously estimate the number of clusters $K$ and the feature-sparsity parameter $\lambda$ in sparse K-means, performing among the best existing estimators for $K$ in low…
desk verdict A useful resampling score for joint K and lambda selection in sparse K-means, but the empirical claims are oversold and the evaluation needs independent tuning and full implementation details. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-subject concordance score $S_i(K)$ defined in Eq. (4): with $T$ the $n \times n$ co-membership matrix from clustering the full data and $\bar{T}^{(sub)}$ the element-wise average of $B$ subsample co-membership matrices, $S_i(K) = (\text{mean of } \bar{T}^{(sub)}_{ij} \text{ over } j \text{ with } T_{ij}=1) + (\text{mean of } (1-\bar{T}^{(sub)}_{ij}) \text{ over } j \text{ with } T_{ij}=0) - 1$. This is the Youden index (sensitivity + specificity − 1) of a dichotomous diagnostic test, with the full-data partition cast as truth. The score becomes a $K$ selector through a trimmed mean $S^*_\rho(K)$ that iteratively removes the lowest-scoring $\rho = 5\%$ of subjects, protecting against scattered points. For sparse K-means, a second Youden-type score $F(K, \lambda)$ checks stability of the selected-feature set, and a two-stage rule first maximizes $S^*$ over $(K, \lambda)$ and then, with $K$ fixed, maximizes $S^* + F$ to choose $\lambda$; a bisection grid over $\lambda$ spaced by log-scaled numbers of selected features keeps the search affordable.
What would settle it
Run S4 on a benchmark with $K = 3$ where full-data K-means has two nearly equal local optima at $K = 3$ (for example, symmetric clusters with label switching across restarts); if $S^*_\rho$ is maximized at a $K$ other than 3 in most replicates, the assumption that the full-data co-membership matrix is a trustworthy reference fails.
Extended reading notes
Core claim
S4's central claim is that the right number of clusters, and in sparse K-means the right sparsity level, are the values at which the full-data partition is most reproducible under subsampling. Formally, for each subject $i$ the score $S_i(K)$ treats the full-data co-membership matrix $T$ as the reference and the average of $B$ subsample co-membership matrices as the test result: it averages the subsample co-membership rates over subjects that $T$ puts in the same cluster as $i$ (sensitivity), averages the subsample non-co-membership rates over subjects that $T$ separates from $i$ (specificity), and subtracts one, the Youden index. The trimmed mean $S^*_\rho(K)$ over the best $100-\rho$ percent of subjects removes scattered points, and $K$ is chosen to maximize it. For sparse K-means, a parallel feature concordance score $F(K, \lambda)$ measures whether the set of nonzero feature weights is also stable, and $\lambda$ is chosen, after $K$ is fixed, by maximizing $S^* + F$. The paper's evidence is simulation across well-separated, overlapping, null, independent-feature and correlated-gene settings, plus nine real datasets; its stated conclusion is that S4 is the only method that generally performs among the top in all low-dimensional settings and universally achieves better clustering accuracy and feature selection in high-dimensional comparisons.
Load-bearing premise
The method's score treats the full-data clustering as the ground truth; if that clustering itself is unstable or trapped in a local optimum, the score measures consistency with an arbitrary reference, and the method's validity depends on K-means at the true $K$ being stable and representative under subsampling.
Editorial extensions
If this is right
- In low-dimensional K-means problems, S4 is claimed to be among the top performers across all ten simulation settings, including overlapping and null-data cases where gap-statistic variants fail.
- In high-dimensional sparse K-means, S4 is claimed to achieve higher adjusted Rand index for clustering, higher Jaccard index for feature selection, and lower root-mean-squared error for $K$ than bivariate gap statistic and prediction strength, with the largest gains when few informative features or weak correlation are present.
- In real applications, S4 recovers the correct or closest $K$ in almost all nine datasets and selects fewer predictive genes than gap statistic or prediction strength at equal or better clustering accuracy.
- With S4, users can estimate $K$ and $\lambda$ in one pass rather than estimating $K$ first and then tuning $\lambda$ by a separate gap-statistic step, removing a subjective two-stage decision.
- The authors argue that the same stability rationale can be carried to other sparse clustering engines, such as sparse Gaussian mixture and sparse Poisson mixture models.
Reading between the lines
- A direct consequence the authors do not spell out: the S4 curve itself is a reproducibility diagnostic, so uniformly low $S^*$ across all $K$ would signal that no stable cluster structure exists, not just that $K$ is hard to choose.
- The two-stage ordering suggests a broader tuning principle for joint discrete and continuous parameters in unsupervised learning, namely to fix the discrete structure by a stability criterion before tuning a continuous penalty, which could transfer to other regularized clustering or latent-variable models.
- Because S4 only requires a co-membership matrix and a selected-feature indicator, it could be tested on other clustering algorithms with built-in sparsity, such as sparse Gaussian or Poisson mixtures, and on non-Euclidean dissimilarities, even though the paper only demonstrates K-means and sparse K-means.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes S4, a resampling-based stability score for estimating the number of clusters K in K-means and, in sparse K-means, for simultaneously estimating K and the feature-sparsity parameter λ. The score compares the full-data co-membership matrix T with the averaged co-membership matrix over repeated subsamples, defining per-subject sensitivity and specificity terms (a Youden index) and then averaging after trimming the lowest ρ% of subject scores. For simultaneous (K, λ) selection, a feature-selection concordance score F is added, with K chosen from the clustering score alone and λ chosen from the sum of the clustering and feature scores. The authors also extend the gap statistic and prediction strength to the joint (K, λ) setting, evaluate S4 on ten low-dimensional K-means settings, on sparse K-means simulations with independent and correlated features, and on nine real datasets. The paper claims that S4 performs among the best in estimating K and that it outperforms the extended gap statistic and prediction strength in high-dimensional clustering and feature selection.
Significance. The topic is timely and the claimed contribution is useful: a single automatic choice of K and λ for sparse K-means would fill a real gap, and the paper provides a concrete criterion, an efficient bisection grid for λ, extensions of two benchmark methods, and an unusually broad simulation and real-data evaluation. If the empirical claims withstand scrutiny, S4 could become a practical default for exploratory high-dimensional clustering. I also credit the authors for explicitly discussing two limitations and for reporting runtime information. However, the central comparative claim is weakened by post hoc selection of the trimming parameter ρ on the same simulation settings later used for evaluation, by an unspecified subsampling fraction f, by an unsupported null threshold s0, and by at least one headline statement that the paper's own tables contradict. The conceptual reliance of the score on treating the full-data clustering as truth is acknowledged in the text but its implications for the method's validity are not addressed. No code or reproducibility package is provided, which matters for a method whose practical use depends on several free parameters.
major comments (5)
- [§5.1, Tables 2 and 3] The choice ρ = 5 is selected on the same ten simulation settings that are later used to evaluate S4. The text states that Table 2 is a sensitivity analysis of ρ on settings 1–10 and then says 'we set ρ = 5 in all comparisons' and 'throughout this paper'; Table 3 then reports S4's performance on the same ten settings with ρ = 5. The comparison in Table 3 is therefore post-selection: the reported advantage of S4 may reflect tuning to the benchmark rather than a property of a fixed method. The authors should either validate ρ on independent settings (for example, by a nested split of the simulation settings into tuning and evaluation sets), report results across ρ to show that the qualitative conclusions are insensitive, or clearly present ρ as a tuned parameter with the associated optimism in the reported numbers.
- [§5.1, Table 3 and Section 1] The claim that 'S4 is the only method that generally performs among the top in all settings' is not supported by the paper's own Table 3. In setting 4, S4 selects the correct K in 78 of 100 runs while H, KL, and Gap/Unif select correct K in 98, 94, and 93 runs respectively; in setting 5, S4's 91 is below Gap/PCA, CH, FW, and LD, all at 100; in setting 8, S4's 79 is below Jump (90), silhouette (93), and FW (94); and in setting 10, S4's 4 is the second-worst, above only H's 0 and below PS (15) and LD (16). The authors should either revise this claim to a more precise statement about overall performance or define explicitly what 'among the top' means and show that S4 satisfies that definition in every row.
- [§3, Step 1 and all simulations] The subsampling fraction f is never specified. Step 1 states only that subsamples have size f×n with 0 < f < 1, and no simulation or real-data section reports the value of f used. Since the S4 score in Eq. (4), the trimmed mean, and the feature concordance score all depend on f, the results are not reproducible as stated. The authors should report the f used in every experiment and provide a sensitivity analysis for f, given that the subsample size directly affects the stability of the co-membership estimates.
- [§3, Step 5 and Table 3, setting 1] The null-data threshold s0 = 0.8 is asserted without any sensitivity analysis. The algorithm returns K = 1 when Smax < s0, and the null-detection performance in setting 1 of Table 3 depends directly on this number. The choice of s0 is not derived from data or theory, and no results are shown for nearby thresholds. The authors should either justify s0 empirically across settings or report the sensitivity of the null-detection and K-selection results to s0.
- [§3, Eq. (4) and Section 7] The S4 score treats the full-data clustering co-membership matrix T as the underlying truth when defining sensitivity and specificity. As the paper acknowledges, this is an internal reference for a stability heuristic, but the practical validity of S4 as an estimator of the true K relies on the assumption that K-means at the candidate K is stable and representative under subsampling. If the full-data clustering is itself a local optimum or an unstable partition, the score measures self-consistency with an arbitrary reference. The authors should state this assumption more carefully and provide a diagnostic or additional experiment, for example, comparing S4's selected K when the full-data clustering is obtained from multiple random restarts, to show that the score is not driven by a single unstable K-means run.
minor comments (6)
- [§5.1, paragraph after Table 3] The sentence 'silhouette performs better in settings 6 ∼10 but not settings 6∼10' is self-contradictory; the second range should presumably be settings 1–5.
- [§4.2, S4 paragraph] The text says Si(K, λ) is calculated 'by equation (3) in Section 3', but the subject concordance score is defined in Eq. (4); Eq. (3) defines only the co-membership matrix T. Please correct the cross-reference.
- [§5.3, Simulation III, step 2] The abbreviation 'POI (20)' is never defined; if it denotes a Poisson distribution, please write Poisson(20) or define the abbreviation.
- [Throughout] There are numerous typographical errors that should be corrected in revision, including 'interger' (Step 1), 'caluclate' (§4.2), 'nave' (§4.2), 'noth' (§6.2.4), 'timmed' (Supplement captions), and 'SConsidering' in the S4 name in Section 1.
- [§5.1, text before Table 2] The reference to 'toy examples in Figure ??' is unresolved; it should point to Figure S2 or another specific figure.
- [General reproducibility] No code, package, or data-availability statement is provided. Given the method's reliance on B, f, ρ, s0, and the λ-grid construction, a reproducible implementation would substantially strengthen the paper.
Circularity Check
S4's low-dimensional benchmark advantage is partly post-selection: the trimming hyperparameter rho was tuned on the same ten simulation settings later used to declare S4 the top method.
-
fitted input called prediction
[Section 5.1, 'Sensitivity analysis for trimmed mean in S4' paragraph and 'Comparison of ten methods in ten simulation settings' paragraph, Tables 2 and 3]
"we perform a sensitivity analysis of different trimming parameter ρ = 0, 2, 5, 8, 10, 15, 20 by applying S4 to 10 simulation settings. We find that ρ = 2 ∼ 10 generally provide good performance and ρ = 5 seems to give the best overall estimation. Throughout this paper, we set ρ = 5 in all comparisons. ... S4 is the only method that generally performs among the top in all settings."
Table 2 selects rho by inspecting S4's correct-K counts on the ten simulation settings, and the text then fixes rho = 5 'in all comparisons' because it gives the best overall estimation. Table 3 evaluates S4 against nine competitors on exactly the same ten settings with this chosen rho and uses those numbers to claim that S4 is 'the only method that generally performs among the top in all settings.' The benchmark verdict is therefore post-selection: the hyperparameter was tuned to the very cases used as evidence of superiority. The low-dimensional performance claim is not an independent evaluation of a fixed method, so the reported 'top performer' result is partly forced by the preceding sensitivity fit.
full rationale
The paper's core S4 score is a stability heuristic: it measures concordance between full-data K-means comembership T and averaged subsample comembership, treating T as an internal reference. That is a self-consistency criterion, not a circular derivation; K and lambda are not defined in terms of the fitted score in a way that makes the estimates equal to the inputs by construction. The high-dimensional sparse K-means simulations (Tables 4 and 5) and the nine real-data applications provide independent evidence for S4, and those comparisons are not directly tuned on the same response variable. However, the low-dimensional claim in Table 3 is weakened because the trimming proportion rho was chosen by inspecting correct-K counts on the same ten simulation settings used for the method comparison. This is a validation circularity rather than an equation-level equivalence. The self-citation to Tseng (2007) for the scattered-point motivation is not load-bearing because the paper supplies its own sensitivity analysis for trimming. Overall, no derivation reduces to its inputs, so the score is modest.
Assumptions & free parameters
free parameters (3)
- Trimming proportion rho =
5%
- Null threshold s0 =
0.8
- Subsampling fraction f =
not specified in paper
assumptions (4)
- domain assumption Pairwise co-membership stability under random subsampling is a valid proxy for the true number of clusters.
- domain assumption The full-data comembership matrix T(K) can be treated as the underlying truth when computing per-subject sensitivity and specificity.
- domain assumption Sparse K-means with the L1-constrained weight objective of Witten and Tibshirani (2010) is a valid clustering model whose λ controls feature sparsity.
- domain assumption K-means returns a representative full-data and subsample clustering; local optima and initialization are not separately addressed.
Cite this review
Pith. "Pith review of Simultaneous Estimation of Number of Clusters and Feature Sparsity in Clustering High-Dimensional Data." pith.science (2026). https://pith.science/paper/5JA5PVES
@misc{pith2026190901930,
author = {Pith},
title = {Pith review of: Simultaneous Estimation of Number of Clusters and Feature Sparsity in Clustering High-Dimensional Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/5JA5PVES}},
note = {Machine review of arXiv:1909.01930}
}
read the original abstract
Estimating the number of clusters (K) is a critical and often difficult task in cluster analysis. Many methods have been proposed to estimate K, including some top performers using resampling approach. When performing cluster analysis in high-dimensional data, simultaneous clustering and feature selection is needed for improved interpretation and performance. To our knowledge, none has investigated simultaneous estimation of K and feature selection in an exploratory cluster analysis. In this paper, we propose a resampling method to meet this gap and evaluate its performance under the sparse K-means clustering framework. The proposed target function balances between sensitivity and specificity of clustering evaluation of pairwise subjects from clustering of full and subsampled data. Through extensive simulations, the method performs among the best over classical methods in estimating K in low-dimensional data. For high-dimensional simulation data, it also shows superior performance to simultaneously estimate K and feature sparsity parameter. Finally, we evaluated the methods in four microarray, two RNA-seq, one SNP and two non-omics datasets. The proposed method achieves better clustering accuracy with fewer selected predictive genes in almost all real applications.
Reference graph
Works this paper leans on
-
[1]
Balgobind, B. V., Van den Heuvel-Eibrink, M. M., Menezes, R. X., Reinhardt, D., Hollink, I. H., Peters, S. T., van Wering, E. R., Kaspers, G. J., Cloos, J., de Bont, E. S. M., et al. (2010). Evaluation of gene expression signatures predictive for cytogenetic and molecular subtypes of pediatric acute myeloid leukemia. haematologica , pages haematol--2010
work page 2010
-
[2]
Banfield, J. D. and Raftery, A. E. (1993). Model-based gaussian and non-gaussian clustering. Biometrics , pages 803--821
1993
-
[3]
Ben-Hur, A., Elisseeff, A., and Guyon, I. (2001). A stability based method for discovering structure in clustered data. In Biocomputing 2002 , pages 6--17. World Scientific
work page 2001
-
[4]
and Harabasz, J
Cali \'n ski, T. and Harabasz, J. (1974). A dendrite method for cluster analysis. Communications in Statistics-theory and Methods , 3(1):1--27
1974
-
[5]
Charrad, M., Ghazzali, N., Boiteau, V., Niknafs, A., and Charrad, M. M. (2014). Package 'nbclust'. Journal of Statistical Software , 61:1--36
work page 2014
-
[6]
Dheeru, D. and Karra Taniskidou, E. (2017). UCI machine learning repository
work page 2017
-
[7]
Dudoit, S. and Fridlyand, J. (2002). A prediction-based resampling method for estimating the number of clusters in a dataset. Genome biology , 3(7):research0036--1
work page 2002
-
[8]
and Wang, J
Fang, Y. and Wang, J. (2012). Selection of the number of clusters via the bootstrap method. Computational Statistics & Data Analysis , 56(3):468--477
2012
Show all 36 references
-
[9]
Hartigan, J. A. (1975). Clustering algorithms, new york: John willey and sons. Inc. Pages113129
1975
-
[10]
and Arabie, P
Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of classification , 2(1):193--218
1985
-
[11]
Jaccard, P. (1901). Distribution de la flore alpine dans le bassin des dranses et dans quelques r \'e gions voisines. Bull Soc Vaudoise Sci Nat , 37:241--272
1901
-
[12]
Johnson, S. C. (1967). Hierarchical clustering schemes. Psychometrika , 32(3):241--254
1967
-
[13]
and Rousseeuw, P
Kaufman, L. and Rousseeuw, P. (1987). Clustering by means of medoids . North-Holland
1987
-
[14]
J., Rassenti, L
Kohlmann, A., Kipps, T. J., Rassenti, L. Z., Downing, J. R., Shurtleff, S. A., Mills, K. I., Gilkes, A. F., Hofmann, W.-K., Basso, G., Dell’Orto, M. C., et al. (2008). An international standardization programme towards the application of gene expression profiling in routine le...
2008
-
[15]
Krzanowski, W. J. and Lai, Y. (1988). A criterion for determining the number of groups in a data set using sum-of-squares clustering. Biometrics , pages 23--34
1988
-
[16]
L., and Buhmann, J
Lange, T., Roth, V., Braun, M. L., and Buhmann, J. M. (2004). Stability-based validation of clustering solutions. Neural computation , 16(6):1299--1323
2004
-
[17]
and Domany, E
Levine, E. and Domany, E. (2001). Resampling method for unsupervised estimation of cluster validity. Neural computation , 13(11):2573--2593
2001
-
[18]
D., Cao, J., Wang, S., Wang, J., Sarkar, S., Vigorito, M., Ma, J
Li, M. D., Cao, J., Wang, S., Wang, J., Sarkar, S., Vigorito, M., Ma, J. Z., and Chang, S. L. (2013). Transcriptome sequencing of gene expression in the brain of the hiv-1 transgenic rat. PLoS One , 8(3):e59582
2013
-
[19]
MacQueen, J. et al. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability , volume 1, pages 281--297. Oakland, CA, USA
1967
-
[20]
and Ramler, I
Maitra, R. and Ramler, I. P. (2009). Clustering in the presence of scatter. Biometrics , 65(2):341--352
2009
-
[21]
Mallah, C., Cope, J., and Orwell, J. (2013). Plant leaf classification using probabilistic integration of shape, texture and margin features. Signal Processing, Pattern Recognition and Applications , 5(1)
2013
-
[22]
Milligan, G. W. and Cooper, M. C. (1985). An examination of procedures for determining the number of clusters in a data set. Psychometrika , 50(2):159--179
1985
-
[23]
and Shen, X
Pan, W. and Shen, X. (2007). Penalized model-based clustering with application to variable selection. Journal of Machine Learning Research , 8(May):1145--1164
2007
-
[24]
Rousseeuw, P. J. (1987). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics , 20:53--65
1987
-
[25]
I., Cooke, M
Su, A. I., Cooke, M. P., Ching, K. A., Hakak, Y., Walker, J. R., Wiltshire, T., Orth, A. P., Vega, R. G., Sapinoso, L. M., Moqrich, A., et al. (2002). Large-scale analysis of the human and mouse transcriptomes. Proceedings of the National Academy of Sciences , 99(7):4465--4470
2002
-
[26]
Sugar, C. A. and James, G. M. (2003). Finding the number of clusters in a dataset: An information-theoretic approach. Journal of the American Statistical Association , 98(463):750--763
2003
-
[27]
and Walther, G
Tibshirani, R. and Walther, G. (2005). Cluster validation by prediction strength. Journal of Computational and Graphical Statistics , 14(3):511--528
2005
-
[28]
Tibshirani, R., Walther, G., and Hastie, T. (2001). Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 63(2):411--423
2001
-
[29]
Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., and Altman, R. B. (2001). Missing value estimation methods for dna microarrays. Bioinformatics , 17(6):520--525
2001
-
[30]
Tseng, G. C. (2007). Penalized and weighted k-means for clustering with scattered objects and prior information in high-throughput biological data. Bioinformatics , 23(17):2247--2255
2007
-
[31]
G., Wouters, B
Verhaak, R. G., Wouters, B. J., Erpelinck, C. A., Abbas, S., Beverloo, H. B., Lugthart, S., L \"o wenberg, B., Delwel, R., and Valk, P. J. (2009). Prediction of molecular subtypes in acute myeloid leukemia based on gene expression profiling. haematologica , 94(1):131--134
2009
-
[32]
Witten, D. M. et al. (2011). Classification and clustering of sequencing data using a poisson model. The Annals of Applied Statistics , 5(4):2493--2518
2011
-
[33]
Witten, D. M. and Tibshirani, R. (2010). A framework for feature selection in clustering. Journal of the American Statistical Association , 105(490):713--726
2010
-
[34]
Witten, D. M. and Tibshirani, R. (2013). sparcl: Perform sparse hierarchical clustering and sparse k-means clustering. R package version , 1(3)
2013
-
[35]
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer , 3(1):32--35
1950
-
[36]
Zhou, H., Pan, W., and Shen, X. (2009). Penalized model-based clustering with unconstrained covariance matrices. Electronic journal of statistics , 3:1473
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.