{"id":"fc7f11ed-9fba-4d9f-bb4c-918a9313e5ab","arxiv_id":"1909.01930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new subsampling score, S4, selects the number of clusters and the feature-sparsity parameter together in sparse K-means and outperforms gap-statistic and prediction-strength extensions in simulations and nine real datasets.","lead":"This paper introduces a stability-based method that automatically chooses both the number of clusters and how many features to keep when clustering high-dimensional data. Scientists analyzing gene-expression or other wide datasets may use it to obtain more accurate clusters with fewer selected genes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"S4's reported advantage is partly an artifact of tuning the trimming parameter rho on the same simulation settings used for evaluation; the robustness claim lacks an independent test.","rationale":"The reader's weakest assumption (using the full-data comembership matrix T as truth) is a valid conceptual limitation shared by stability-based methods, but it is not the most load-bearing issue for this paper's central claim. The paper's evidence for superiority is primarily empirical, and that evidence is compromised by a classic selection-on-the-test-set problem: rho = 5 was chosen from a sensitivity analysis on the very same ten simulation settings that are then used to demonstrate S4's performance. The omission of f and the absence of code make the results irreproducible, and the absence of a sensitivity analysis for s0 leaves the null-data decision rule unvalidated. These issues can be resolved by a nested tuning experiment and by reporting results across rho and f on a fresh simulation family. If S4 remains among the top after such a check, the conditional acceptance is warranted; if not, the strength of the paper's claims should be reduced. I therefore keep the reader's CONDITIONAL verdict unchanged rather than either accepting or rejecting outright.","tokens_in":22529,"tokens_out":10520,"duration_ms":116187,"concrete_test":"Run a nested tuning experiment: for each of the ten Simulation I settings, tune rho (and ideally s0) on the other nine settings, then evaluate S4 on the held-out setting, and repeat over all ten folds; compare the average number of correct K choices with Table 3. Additionally, generate a new simulation family not used in the paper (e.g., five clusters, unequal sizes, n=200, p=2000, correlated features, varying overlap) and run S4 with rho = 0, 2, 5, 10 and f = 0.5, 0.7, 0.9, plus extended Gap and PS. If S4 with the tuned rho is not consistently among the top, or if its advantage over Gap/PS disappears for some (rho, f), the robustness claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that S4 'generally performs among the top in all settings' and 'universally achieves better' clustering accuracy in high-dimensional settings—rests on simulations in which a key hyperparameter was selected on the same data used for the comparison. Section 5.1, Table 2 is a sensitivity analysis for the trimming proportion rho across the ten simulation settings, and the text then says 'we set rho = 5 in all comparisons' 'throughout this paper.' Those same ten settings appear in Table 3, where S4 is declared the only method generally in the top. The reported performance numbers are therefore post-selection: rho was chosen to maximize S4's success on the benchmark, so the comparison is not a fair evaluation of a fixed method. In addition, the subsampling fraction f is never specified (Step 1 says only 0 < f < 1), and the threshold s0 = 0.8 is asserted without sensitivity analysis. Because S4's score in Eq. (4) and the trimming step depend on both f and rho, the results are not reproducible as stated and may not generalize to settings with different cluster overlap, dimensionality, or feature correlation. The T-as-truth assumption is a real conceptual limitation, but the more immediate threat to the central claim is that the headline numbers were tuned on the test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S4, a resampling-based stability score for estimating the number of clusters K in K-means and, in sparse K-means, for simultaneously estimating K and the feature-sparsity parameter λ. The score compares the full-data co-membership matrix T with the averaged co-membership matrix over repeated subsamples, defining per-subject sensitivity and specificity terms (a Youden index) and then averaging after trimming the lowest ρ% of subject scores. For simultaneous (K, λ) selection, a feature-selection concordance score F is added, with K chosen from the clustering score alone and λ chosen from the sum of the clustering and feature scores. The authors also extend the gap statistic and prediction strength to the joint (K, λ) setting, evaluate S4 on ten low-dimensional K-means settings, on sparse K-means simulations with independent and correlated features, and on nine real datasets. The paper claims that S4 performs among the best in estimating K and that it outperforms the extended gap statistic and prediction strength in high-dimensional clustering and feature selection.","tokens_in":22725,"tokens_out":4834,"duration_ms":51716,"significance":"The topic is timely and the claimed contribution is useful: a single automatic choice of K and λ for sparse K-means would fill a real gap, and the paper provides a concrete criterion, an efficient bisection grid for λ, extensions of two benchmark methods, and an unusually broad simulation and real-data evaluation. If the empirical claims withstand scrutiny, S4 could become a practical default for exploratory high-dimensional clustering. I also credit the authors for explicitly discussing two limitations and for reporting runtime information. However, the central comparative claim is weakened by post hoc selection of the trimming parameter ρ on the same simulation settings later used for evaluation, by an unspecified subsampling fraction f, by an unsupported null threshold s0, and by at least one headline statement that the paper's own tables contradict. The conceptual reliance of the score on treating the full-data clustering as truth is acknowledged in the text but its implications for the method's validity are not addressed. No code or reproducibility package is provided, which matters for a method whose practical use depends on several free parameters.","major_comments":[{"comment":"The choice ρ = 5 is selected on the same ten simulation settings that are later used to evaluate S4. The text states that Table 2 is a sensitivity analysis of ρ on settings 1–10 and then says 'we set ρ = 5 in all comparisons' and 'throughout this paper'; Table 3 then reports S4's performance on the same ten settings with ρ = 5. The comparison in Table 3 is therefore post-selection: the reported advantage of S4 may reflect tuning to the benchmark rather than a property of a fixed method. The authors should either validate ρ on independent settings (for example, by a nested split of the simulation settings into tuning and evaluation sets), report results across ρ to show that the qualitative conclusions are insensitive, or clearly present ρ as a tuned parameter with the associated optimism in the reported numbers.","section":"§5.1, Tables 2 and 3"},{"comment":"The claim that 'S4 is the only method that generally performs among the top in all settings' is not supported by the paper's own Table 3. In setting 4, S4 selects the correct K in 78 of 100 runs while H, KL, and Gap/Unif select correct K in 98, 94, and 93 runs respectively; in setting 5, S4's 91 is below Gap/PCA, CH, FW, and LD, all at 100; in setting 8, S4's 79 is below Jump (90), silhouette (93), and FW (94); and in setting 10, S4's 4 is the second-worst, above only H's 0 and below PS (15) and LD (16). The authors should either revise this claim to a more precise statement about overall performance or define explicitly what 'among the top' means and show that S4 satisfies that definition in every row.","section":"§5.1, Table 3 and Section 1"},{"comment":"The subsampling fraction f is never specified. Step 1 states only that subsamples have size f×n with 0 < f < 1, and no simulation or real-data section reports the value of f used. Since the S4 score in Eq. (4), the trimmed mean, and the feature concordance score all depend on f, the results are not reproducible as stated. The authors should report the f used in every experiment and provide a sensitivity analysis for f, given that the subsample size directly affects the stability of the co-membership estimates.","section":"§3, Step 1 and all simulations"},{"comment":"The null-data threshold s0 = 0.8 is asserted without any sensitivity analysis. The algorithm returns K = 1 when Smax < s0, and the null-detection performance in setting 1 of Table 3 depends directly on this number. The choice of s0 is not derived from data or theory, and no results are shown for nearby thresholds. The authors should either justify s0 empirically across settings or report the sensitivity of the null-detection and K-selection results to s0.","section":"§3, Step 5 and Table 3, setting 1"},{"comment":"The S4 score treats the full-data clustering co-membership matrix T as the underlying truth when defining sensitivity and specificity. As the paper acknowledges, this is an internal reference for a stability heuristic, but the practical validity of S4 as an estimator of the true K relies on the assumption that K-means at the candidate K is stable and representative under subsampling. If the full-data clustering is itself a local optimum or an unstable partition, the score measures self-consistency with an arbitrary reference. The authors should state this assumption more carefully and provide a diagnostic or additional experiment, for example, comparing S4's selected K when the full-data clustering is obtained from multiple random restarts, to show that the score is not driven by a single unstable K-means run.","section":"§3, Eq. (4) and Section 7"}],"minor_comments":[{"comment":"The sentence 'silhouette performs better in settings 6 ∼10 but not settings 6∼10' is self-contradictory; the second range should presumably be settings 1–5.","section":"§5.1, paragraph after Table 3"},{"comment":"The text says Si(K, λ) is calculated 'by equation (3) in Section 3', but the subject concordance score is defined in Eq. (4); Eq. (3) defines only the co-membership matrix T. Please correct the cross-reference.","section":"§4.2, S4 paragraph"},{"comment":"The abbreviation 'POI (20)' is never defined; if it denotes a Poisson distribution, please write Poisson(20) or define the abbreviation.","section":"§5.3, Simulation III, step 2"},{"comment":"There are numerous typographical errors that should be corrected in revision, including 'interger' (Step 1), 'caluclate' (§4.2), 'nave' (§4.2), 'noth' (§6.2.4), 'timmed' (Supplement captions), and 'SConsidering' in the S4 name in Section 1.","section":"Throughout"},{"comment":"The reference to 'toy examples in Figure ??' is unresolved; it should point to Figure S2 or another specific figure.","section":"§5.1, text before Table 2"},{"comment":"No code, package, or data-availability statement is provided. Given the method's reliance on B, f, ρ, s0, and the λ-grid construction, a reproducible implementation would substantially strengthen the paper.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: S4 is a reasonable, potentially useful stability heuristic, and the two-stage K-then-lambda design plus the feature-concordance term is a genuine contribution. The real-data results—fewer selected genes with equal or better ARI on several omics sets—are the most convincing part. But the paper as posted overstates its own performance, and the main empirical comparison is weakened by exactly the kind of post-selection that the stress-test flags.\n\nWhat's new: the S4 score (per-subject sensitivity and specificity against the full-data comembership, iteratively trimmed) and the extension to sparse K-means with a feature-selection concordance term. The bisection grid for lambda is a nice practical idea. The benchmark across six simulation settings with correlated genes and nine real datasets is broad and useful, even if not as clean as they claim.\n\nThe soft spots, in order of importance. First, the trimming constant rho=5 is chosen from a sensitivity table on the same ten simulation settings that later appear in the head-to-head comparison (Tables 2 and 3). That makes the \"S4 is the only method generally among the top\" line a post-selection result, not an honest out-of-sample claim. They should pick rho on a separate tuning set or at least show the qualitative pattern holds across all reasonable rho. Second, the subsampling fraction f is never specified; the algorithm says only 0 < f < 1. No method can be reproduced or checked without that number, and missing code makes this worse. Third, the universal claims are not supported by their own tables: in Table 3, S4 gets 4/100 on setting 10 and 91/100 on setting 5, while several other methods do better; the \"only method\" sentence is simply false. In Tables 4 and 5, S4 is usually best but not always—PS beats it on ARI in one correlated setting, and the RMSE differences are sometimes negligible. Softening the language would make the paper more credible, not less.\n\nThe T-as-truth concern is real but minor. Any stability method needs a reference, and using the full-data clustering as that reference is a defensible heuristic. The circularity burden is low; the method is not rigged to output the truth. The bigger issue is the evaluation hygiene.\n\nBottom line: I'd send this to peer review. The idea is worth serious referee time, and with code, a fixed f, an honest reading of the tables, and a sensitivity analysis on a separate set, it could become a standard tool. As it stands, it is a solid draft, not a finished paper.\n\nRegards,\n[Name]","headline":"A useful resampling score for joint K and lambda selection in sparse K-means, but the empirical claims are oversold and the evaluation needs independent tuning and full implementation details.","tokens_in":23301,"tokens_out":4073,"would_cite":true,"duration_ms":40845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single resampling score, S4, can simultaneously estimate the number of clusters $K$ and the feature-sparsity parameter $\\lambda$ in sparse K-means, performing among the best existing estimators for $K$ in low…","keywords":["number of clusters","feature selection","sparse K-means","resampling","stability","Youden index","high-dimensional clustering","K-means"],"falsifier":"Run S4 on a benchmark with $K = 3$ where full-data K-means has two nearly equal local optima at $K = 3$ (for example, symmetric clusters with label switching across restarts); if $S^*_\\rho$ is maximized at a $K$ other than 3 in most replicates, the assumption that the full-data co-membership matrix is a trustworthy reference fails.","tokens_in":22279,"feed_emoji":"📊","tokens_out":12590,"duration_ms":103874,"temperature":0.7,"pith_summary":"The paper proposes S4, a resampling-based score that estimates the number of clusters $K$ for K-means and, in a two-stage extension, estimates both $K$ and the feature-sparsity parameter $\\lambda$ for sparse K-means. The score judges a candidate $(K, \\lambda)$ by how well repeated subsample clusterings agree with the full-data clustering, combining sensitivity and specificity into a per-subject Youden index and trimming the most unstable subjects. The authors' central claim is that this one procedure performs among the best existing $K$ estimators across ten low-dimensional simulation settings, and outperforms bivariate gap statistic and prediction strength at joint $K$-and-$\\lambda$ estimation in high-dimensional simulations with independent and correlated features. On nine real datasets (microarray, RNA-seq, SNP, and two non-omics sets), they report that S4 recovers the correct or closest cluster number in almost all cases while selecting fewer predictive genes at equal or better clustering accuracy. If true, this gives practitioners a single automatic choice of $K$ and $\\lambda$ instead of the current two-step practice of estimating $K$ and then tuning $\\lambda$ separately.","feed_headline":"One resampling score picks cluster count and feature sparsity together","feed_subtitle":"The S4 method matches or beats two-step tuning on omics and non-omics data, selecting fewer predictive genes.","key_machinery":"The load-bearing object is the per-subject concordance score $S_i(K)$ defined in Eq. (4): with $T$ the $n \\times n$ co-membership matrix from clustering the full data and $\\bar{T}^{(sub)}$ the element-wise average of $B$ subsample co-membership matrices, $S_i(K) = (\\text{mean of } \\bar{T}^{(sub)}_{ij} \\text{ over } j \\text{ with } T_{ij}=1) + (\\text{mean of } (1-\\bar{T}^{(sub)}_{ij}) \\text{ over } j \\text{ with } T_{ij}=0) - 1$. This is the Youden index (sensitivity + specificity − 1) of a dichotomous diagnostic test, with the full-data partition cast as truth. The score becomes a $K$ selector through a trimmed mean $S^*_\\rho(K)$ that iteratively removes the lowest-scoring $\\rho = 5\\%$ of subjects, protecting against scattered points. For sparse K-means, a second Youden-type score $F(K, \\lambda)$ checks stability of the selected-feature set, and a two-stage rule first maximizes $S^*$ over $(K, \\lambda)$ and then, with $K$ fixed, maximizes $S^* + F$ to choose $\\lambda$; a bisection grid over $\\lambda$ spaced by log-scaled numbers of selected features keeps the search affordable.","core_discovery":"S4's central claim is that the right number of clusters, and in sparse K-means the right sparsity level, are the values at which the full-data partition is most reproducible under subsampling. Formally, for each subject $i$ the score $S_i(K)$ treats the full-data co-membership matrix $T$ as the reference and the average of $B$ subsample co-membership matrices as the test result: it averages the subsample co-membership rates over subjects that $T$ puts in the same cluster as $i$ (sensitivity), averages the subsample non-co-membership rates over subjects that $T$ separates from $i$ (specificity), and subtracts one, the Youden index. The trimmed mean $S^*_\\rho(K)$ over the best $100-\\rho$ percent of subjects removes scattered points, and $K$ is chosen to maximize it. For sparse K-means, a parallel feature concordance score $F(K, \\lambda)$ measures whether the set of nonzero feature weights is also stable, and $\\lambda$ is chosen, after $K$ is fixed, by maximizing $S^* + F$. The paper's evidence is simulation across well-separated, overlapping, null, independent-feature and correlated-gene settings, plus nine real datasets; its stated conclusion is that S4 is the only method that generally performs among the top in all low-dimensional settings and universally achieves better clustering accuracy and feature selection in high-dimensional comparisons.","pith_inferences":["A direct consequence the authors do not spell out: the S4 curve itself is a reproducibility diagnostic, so uniformly low $S^*$ across all $K$ would signal that no stable cluster structure exists, not just that $K$ is hard to choose.","The two-stage ordering suggests a broader tuning principle for joint discrete and continuous parameters in unsupervised learning, namely to fix the discrete structure by a stability criterion before tuning a continuous penalty, which could transfer to other regularized clustering or latent-variable models.","Because S4 only requires a co-membership matrix and a selected-feature indicator, it could be tested on other clustering algorithms with built-in sparsity, such as sparse Gaussian or Poisson mixtures, and on non-Euclidean dissimilarities, even though the paper only demonstrates K-means and sparse K-means."],"forward_implications":["In low-dimensional K-means problems, S4 is claimed to be among the top performers across all ten simulation settings, including overlapping and null-data cases where gap-statistic variants fail.","In high-dimensional sparse K-means, S4 is claimed to achieve higher adjusted Rand index for clustering, higher Jaccard index for feature selection, and lower root-mean-squared error for $K$ than bivariate gap statistic and prediction strength, with the largest gains when few informative features or weak correlation are present.","In real applications, S4 recovers the correct or closest $K$ in almost all nine datasets and selects fewer predictive genes than gap statistic or prediction strength at equal or better clustering accuracy.","With S4, users can estimate $K$ and $\\lambda$ in one pass rather than estimating $K$ first and then tuning $\\lambda$ by a separate gap-statistic step, removing a subjective two-stage decision.","The authors argue that the same stability rationale can be carried to other sparse clustering engines, such as sparse Gaussian mixture and sparse Poisson mixture models."],"supporting_citations":[{"why":"Supplies the K-means clustering engine whose number of clusters and sparsity parameter S4 estimates.","marker":"(MacQueen et al., 1967)"},{"why":"Supplies the sparse K-means objective with L1 feature weights that S4 tunes, and the SNP benchmark used in the real applications.","marker":"(Witten and Tibshirani, 2010)"},{"why":"Supplies the gap statistic baseline and the well-separated simulation settings that S4 is compared against.","marker":"(Tibshirani et al., 2001)"},{"why":"Supplies the prediction strength baseline and the threshold rule S4 borrows for detecting K = 1.","marker":"(Tibshirani and Walther, 2005)"},{"why":"Supplies the Youden index form (sensitivity + specificity − 1) used to define the per-subject concordance score.","marker":"(Youden, 1950)"},{"why":"Motivates the treatment of scattered subjects that leads S4 to trim the lowest concordance scores.","marker":"(Tseng, 2007)"},{"why":"Further documents the scattered-subject problem that the trimmed mean in S4 is designed to address.","marker":"(Maitra and Ramler, 2009)"}],"fun_headline_variants":["One criterion picks cluster count and sparsity","Resampling yields K and sparsity simultaneously","Joint estimation of clusters and feature sparsity","Single stability score tunes K and lambda","S4: simultaneous cluster count and sparsity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's score treats the full-data clustering as the ground truth; if that clustering itself is unstable or trapped in a local optimum, the score measures consistency with an arbitrary reference, and the method's validity depends on K-means at the true $K$ being stable and representative under subsampling.","fun_headline_variants_meta":{"raw":{"variants":["One criterion picks cluster count and sparsity","Resampling yields K and sparsity simultaneously","Joint estimation of clusters and feature sparsity","Single stability score tunes K and lambda","S4: simultaneous cluster count and sparsity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1296,"prompt_tokens":1045,"completion_tokens":251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":185}},"tokens_in":661,"tokens_out":251,"duration_ms":3128,"temperature":1.0,"reasoning_tokens":185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:04:41.130039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run S4 on a benchmark with $K = 3$ where full-data K-means has two nearly equal local optima at $K = 3$ (for example, symmetric clusters with label switching across restarts); if $S^*_\\rho$ is maximized at a $K$ other than 3 in most replicates, the assumption that the full-data co-membership matrix is a trustworthy reference fails.","supporting_citations":[{"cited_title":"and Walther, G","cited_arxiv_id":null,"evidence_quote":"Supplies the prediction strength baseline and the threshold rule S4 borrows for detecting K = 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the treatment of scattered subjects that leads S4 to trim the lowest concordance scores."},{"cited_title":"and Ramler, I","cited_arxiv_id":null,"evidence_quote":"Further documents the scattered-subject problem that the trimmed mean in S4 is designed to address."}],"review_version":1}