REVIEW 3 major objections 5 minor 43 references
Towards Truly Unsupervised Evaluation of Feature Selection
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper proposes a truly unsupervised evaluation of feature selection, ranking selectors by the optimal-transport distance between the selected features and the data's top principal components.
desk verdict A clean, label-free feature-selection evaluation idea, honestly framed as a proof of concept; the PCA reference premise and the instability across OT variants are the load-bearing soft spots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the pairing of an unsupervised reference representation with an inter-dataset similarity measure. The reference is Principal Component Analysis: the first $f$ principal components of the full data form the target representation that a good feature subset of size $f$ is expected to resemble. The similarity measure is an optimal-transport distance, instantiated as Earth Mover's Distance, entropic-regularized Sinkhorn, Gromov-Wasserstein, or Sliced Wasserstein, all of which quantify the cost of transforming one distribution into another. The framework's scoring step is the reciprocal transform that turns these distances into similarities, and its ranking step sorts feature-selection algorithms by those similarities. This machinery is what lets the evaluation run without any label information.
What would settle it
Construct a synthetic dataset in which a class label is determined entirely by a single low-variance feature, while all high-variance features are noise. Run the framework with $f=1$ on three selectors: one that picks the low-variance informative feature, one that picks the highest-variance feature, and one random. If the framework ranks the random or high-variance selector above the informative one while the supervised accuracy ranks them oppositely, the claim that closeness to PCA measures feature-selection quality fails for this case.
Extended reading notes
Core claim
The paper's central claim is that feature-selection evaluation can be made completely label-free by comparing the selected feature subspace against a PCA reference. The framework computes, for a fixed number $f$ of features, the optimal-transport distance between the dataset restricted to the $f$ features chosen by a candidate algorithm and the dataset projected onto the first $f$ principal components, then converts the distance into a similarity score (the reciprocal, with infinity meaning identical). The ranking of algorithms follows from these similarities: the more the selected subspace resembles the PCA subspace, the higher it ranks. The paper argues this is “truly unsupervised” because no ground-truth labels enter the scoring at any point, and it uses this framework to show that the widely used clustering-accuracy and NMI evaluations are in fact supervised because they require the true class labels. The paper also finds that the particular optimal-transport measure chosen—Earth Mover's, Sinkhorn, Gromov-Wasserstein, or Sliced Wasserstein—materially changes the resulting rankings.
Load-bearing premise
The scoring assumes that the top principal components of the data are the right reference for feature-selection quality; if the structure that matters for a later task sits in low-variance directions, a selector matching PCA could be ranked above one that keeps the genuinely informative features.
Editorial extensions
If this is right
- The paper's analysis shows that the common “unsupervised” evaluation metrics clustering accuracy and normalized mutual information actually use ground-truth labels and are therefore supervised evaluations run on an unsupervised downstream task.
- The proposed framework ranks feature selectors using only the data and a specified number of principal components, so it can be applied to datasets where no labels exist.
- The choice of optimal-transport distance substantially changes the rankings; in the reported experiments, Gromov-Wasserstein and Sliced Wasserstein correlate positively with label-based performance, while Earth Mover's and Sinkhorn correlate negatively.
- The paper recommends comparing rankings rather than raw optimal-transport scores across different numbers of selected features, because per-budget cost normalization can distort the values.
- The framework is modular and could accept other unsupervised reference representations besides PCA, which the paper identifies as a direction for future work.
Reading between the lines
- A natural extension would be to correct the similarity scores for chance by comparing them with the distribution of scores for random feature subsets, which would make rankings comparable across datasets and budgets.
- If the informative structure for a downstream task lives in low-variance directions, the framework could rank a variance-preserving selector above one that keeps the truly informative features, a failure mode testable on a synthetic dataset with a low-variance class boundary.
- The finding that some optimal-transport measures invert the supervised rankings suggests that each distance encodes a distinct notion of “quality”; the choice should be matched to the intended use of the selected features rather than treated as interchangeable.
- Because Gromov-Wasserstein can compare distributions of different dimensionality, the framework could compare a selected subset directly against the full-dimensional data, removing the need to fix the number of reference components.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that common 'unsupervised' evaluations of feature selection are not truly unsupervised because they use ground-truth labels in downstream clustering metrics such as CLSACC and NMI. It then proposes a label-free framework: for each feature selector that outputs f features, compute an optimal transport distance between the data restricted to those f features and the data projected onto the first f principal components; convert distances to similarities via Eq. (7), and rank selectors by similarity. Experiments on eight high-dimensional datasets compare four OT variants (EMD, Sinkhorn, Gromov-Wasserstein, Sliced Wasserstein) against supervised (ACC, AUC), pseudo-unsupervised (CLSACC, NMI), and model-agnostic (AAD) metrics, reporting Pearson/Spearman correlations and a ranking plot for Isolet. The paper concludes that the framework is a 'truly unsupervised' evaluation approach and that the results are a proof of concept.
Significance. If the framework's central claim holds, it would fill a genuine gap: a label-free, model-agnostic way to rank feature selectors, which is important in domains where ground-truth labels are unavailable. The method has clear strengths: it uses no labels and no fitted parameters; the OT distance is well-defined; and the paper provides a critical discussion of why standard 'unsupervised' metrics such as CLSACC and NMI are not label-free. The authors are also unusually candid in Section 5 about PCA's 'structural limitation' and the proof-of-concept nature of the experiments. However, the validity of the metric is not yet established, because the PCA-as-reference assumption is untested and the empirical support is mixed: two of the four OT variants invert the supervised ranking, aggregate correlations are modest, and no uncertainty quantification is provided. The paper is a promising first step but is substantively incomplete as a demonstration of 'truly unsupervised evaluation of feature selection.'
major comments (3)
- [Section 3 (objective) and Section 5 (Limitations)] The load-bearing premise of the framework is that the top-f PCA representation is a valid reference for feature-selection quality. Section 3 states that 'the objective is to identify feature selection algorithms that select features in a way that they are as close as possible to the representation obtained from PCA,' and Section 5 acknowledges that 'the use of PCA as the reference mechanism imposes a structural limitation.' The manuscript never tests whether closeness to PCA tracks task-relevant information. In particular, a synthetic dataset with a low-variance feature that carries the label signal and high-variance noise features would directly test this: a valid feature-selection quality metric should favor the selector that finds the low-variance feature, while the proposed framework would favor selectors matching PCA's high-variance directions. Without such a test, the central claim that the framework 'evaluates the quality of feature selection algorithms' (Section 3) is not distinguishable from the tautology that it measures agreement with a PCA-based reference.
- [Section 4.2, Figure 4, and Section 5 (Conclusions)] The evidence for predictive validity is mixed and does not support the conclusion that 'the proposed metrics showed consistent correlations with established evaluation metrics.' In Figure 4, OT_EMD2 and OT_SINKHORN2 have Pearson correlations with ACC of -0.02 and -0.03, respectively, and with AUC of 0.07 and 0.06, whereas OT_GW2 and OT_SLICED_SW show correlations of 0.29-0.37 with the supervised block. Because the four instantiations differ only in the OT distance measure, the framework's output is not a single quality measure but a family of measures whose agreement with downstream performance depends on an arbitrary design choice. Moreover, Eq. (7) transforms distances via 1/x; for Pearson correlation this nonlinear transformation changes the coefficient values, so the reported Pearson coefficients are not intrinsic to the OT distance (Spearman, being rank-based, is less affected). The authors should either justify a preferred OT variant a priori, report all results per variant with confidence intervals, or substantially weaken the validity claims.
- [Section 4.1 and Section 4.2 (experimental setup and results)] The empirical basis is too thin for the strength of the central claim: eight datasets, five feature selectors, and no confidence intervals, significance tests, or per-dataset breakdowns. The paper itself describes the results as 'proof of concept' in Section 5, but the abstract and Section 3 present the framework as the solution to 'truly unsupervised evaluation.' The aggregate correlations in Figure 4 could be driven by a small number of datasets, and specific inversions (e.g., Figure 3 shows OT_SLICED_SW ranking Random and SCFS at the top for Isolet) suggest substantial variability. To support the claim that the framework ranks selectors in a way that is useful for practitioners, the authors should report per-dataset and per-feature-fraction correlations, add bootstrap confidence intervals or permutation tests for the aggregate correlations, and ideally include more datasets and selectors. Without this, the evidence for the framework's validity is anecdotal.
minor comments (5)
- [Section 2.2 (AAD description)] The text says 'the eigenvector corresponding to the first principle components'; this should be 'the eigenvector corresponding to the first principal component,' correcting the spelling and the singular/plural mismatch.
- [Figures 2 and 3] Figure 2's caption states 'For all measures, higher is better,' but the text notes that OT_SLICED_SW's resolution degrades; showing rank plots like Figure 3 for all OT variants, not only OT_SLICED_SW, would make the comparison clearer.
- [Figure 4] The heatmaps lack labels for the row and column metric names, which makes the correlation matrix difficult to read; adding axis labels or a legend would improve accessibility.
- [Section 4.2] The phrase 'per-budget cost matrix normalization' is not defined; please specify the normalization formula or cite the FSEVAL implementation so that the reported OT scores are reproducible.
- [References] Reference [31] ('Worse than random') is an arXiv preprint by the same group and is central to the Random baseline; if it is under review, this should be stated, and the relevant code should be made available so readers can reproduce the baseline.
Circularity Check
No significant circularity: the PCA-reference metric is an explicit modeling choice, validated externally against supervised benchmarks rather than derived from them.
full rationale
The paper's derivation chain is self-contained: a feature selector is scored by the optimal transport distance between the data restricted to its f selected features and the data projected onto the first f principal components. This is a fixed, label-free computation with no parameters fitted to the datasets or to any ground-truth labels. The central claim that the framework is truly unsupervised follows from the definition of unsupervised evaluation used in the paper (no label information in the scoring), and the paper itself does not claim that PCA-closeness is interchangeable with downstream task performance; it uses supervised metrics only as external correlation benchmarks after the metric is defined, and explicitly warns that agreement should not be read as construct validity. The load-bearing premise that PCA is a good reference is an acknowledged modeling choice, listed as a structural limitation in Section 5, rather than a result derived from the metric. The paper's self-citations, such as [34] for the claim that only one model-agnostic unsupervised metric exists and [38] for the FSEVAL suite, support background and tooling but are not used to justify the proposed metric's correctness. Thus no step reduces by construction to its own input, and the observed disagreement among OT instantiations is reported as informative rather than hidden. Overall circularity is minimal.
Assumptions & free parameters
assumptions (2)
- domain assumption The first f principal components provide a suitable reference for feature selection quality.
- domain assumption Optimal transport distances between datasets capture the similarity needed for evaluation.
Cite this review
Pith. "Pith review of Towards Truly Unsupervised Evaluation of Feature Selection." pith.science (2026). https://pith.science/paper/PJOJYE7Z
@misc{pith2026260812057,
author = {Pith},
title = {Pith review of: Towards Truly Unsupervised Evaluation of Feature Selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/PJOJYE7Z}},
note = {Machine review of arXiv:2608.12057}
}
read the original abstract
Feature selection is one of the most important and fundamental tasks in data mining, tackled by a family of methods with an established set of evaluation techniques to measure the quality of a specific method. Most of the methods commonly used for the unsupervised evaluation of feature selection algorithms suffer from critical design flaws which question their unsupervised nature. In this paper, we provide a critical discussion on the established allegedly unsupervised evaluation techniques, and shed light on the reasons why they are not truly unsupervised but, at best, supervised evaluation under an unsupervised downstream task. We also propose a novel, truly unsupervised evaluation framework to measure the quality of the feature selection algorithms without any form of information about the labels. The proposed framework utilizes unsupervised Principal Component Analysis, and optimal transport to measure the quality of the feature selection methods in a truly unsupervised manner.
Figures
Reference graph
Works this paper leans on
-
[1]
Amsaleg, L., Bailey, J., Barbe, A., Erfani, S.M., Furon, T., Houle, M.E., Radovanovic, M., Nguyen, X.V.: High intrinsic dimensionality facilitates adver- sarial attack: Theoretical evidence. IEEE Trans. Inf. Forensics Secur.16, 854–865 (2021)
work page 2021
-
[2]
Anderberg, A., Bailey, J., Campello, R.J.G.B., Houle, M.E., Marques, H.O., Radovanovic, M., Zimek, A.: Dimensionality-aware outlier detection. In: SDM. pp. 652–660. SIAM (2024)
work page 2024
-
[3]
Journal of AI and Data Mining9(1), 31–38 (2021)
Asadi Amiri, S., Rajabinasab, M.: Face recognition using color and edge orientation difference histogram. Journal of AI and Data Mining9(1), 31–38 (2021)
work page 2021
-
[4]
Springer (2006), ap- pendix or index reference to Kronecker deltaδij
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer (2006), ap- pendix or index reference to Kronecker deltaδij
work page 2006
- [5]
-
[6]
Bonneel, N., Rabin, J., Peyré, G., Pfister, H.: Sliced and radon wasserstein barycen- ters of measures. J. Math. Imaging Vis.51(1), 22–45 (2015)
work page 2015
-
[7]
Biological Cybernetics59(4-5), 291–294 (1988)
Bourlard, H., Kamp, Y.: Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics59(4-5), 291–294 (1988)
work page 1988
-
[8]
Neurocomputing300, 70–79 (2018)
Cai, J., Luo, J., Wang, S., Yang, S.: Feature selection in machine learning: A new perspective. Neurocomputing300, 70–79 (2018)
2018
Show all 43 references
-
[9]
In: Advances in Neural Information Processing Systems 26 (NIPS 2013) (2013)
Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: Advances in Neural Information Processing Systems 26 (NIPS 2013) (2013)
2013
-
[10]
In: Advances in Neural Information Processing Systems 26
Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: Advances in Neural Information Processing Systems 26. pp. 2292–2300 (2013)
2013
-
[11]
Journal of Machine Learning Research7, 1–30 (2006)
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research7, 1–30 (2006)
2006
-
[12]
IEEE Trans
François, D., Wertz, V., Verleysen, M.: The concentration of fractional distances. IEEE Trans. Knowl. Data Eng.19(7), 873–886 (2007)
2007
-
[13]
Journal of Machine Learning Research3, 1157–1182 (2003)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research3, 1157–1182 (2003)
2003
-
[14]
Hall, M.A.: Correlation-based feature selection for machine learning. Ph.D. thesis, University of Waikato (1999)
1999
-
[15]
Advances in Neural Information Processing Systems18(2005)
He, X., Cai, D., Niyogi, P.: Laplacian score for feature selection. Advances in Neural Information Processing Systems18(2005)
2005
-
[16]
In: NIPS
He, X., Cai, D., Niyogi, P.: Laplacian score for feature selection. In: NIPS. pp. 507–514 (2005)
2005
-
[17]
Houle, M.E., Kriegel, H., Kröger, P., Schubert, E., Zimek, A.: Can shared-neighbor distances defeat the curse of dimensionality? In: SSDBM. pp. 482–500 (2010)
2010
-
[18]
Journal of classification2(1), 193– 218 (1985)
Hubert, L., Arabie, P.: Comparing partitions. Journal of classification2(1), 193– 218 (1985)
1985
-
[19]
Neural Networks166, 188–203 (2023)
Karami, S., Saberi-Movahed, F., Tiwari, P., Marttinen, P., Vahdati, S.: Unsu- pervised feature selection based on variance-covariance subspace distance. Neural Networks166, 188–203 (2023)
2023
-
[20]
Naval Research Logistics Quarterly2(1-2), 83–97 (1955)
Kuhn, H.W.: The Hungarian Method for the assignment problem. Naval Research Logistics Quarterly2(1-2), 83–97 (1955)
1955
-
[21]
ACM Computing Surveys50(6), 94:1–94:45 (2017)
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., Liu, H.: Feature selection: A data perspective. ACM Computing Surveys50(6), 94:1–94:45 (2017)
2017
-
[22]
Journal of Machine Learning Research9(11), 2579–2605 (2008) Towards Truly Unsupervised Evaluation of Feature Selection 15
Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research9(11), 2579–2605 (2008) Towards Truly Unsupervised Evaluation of Feature Selection 15
2008
-
[23]
Journal of Open Source Software3(29), 861 (2018)
McInnes, L., Healy, J., Saul, N., Grossberger, L.: Umap: Uniform manifold approx- imation and projection. Journal of Open Source Software3(29), 861 (2018)
2018
-
[24]
Algorithms14(3), 100 (2021)
Mostert, W., Malan, K.M., Engelbrecht, A.P.: A feature selection algorithm per- formance metric for comparative analysis. Algorithms14(3), 100 (2021)
2021
-
[25]
arXiv preprint arXiv:2501.18901 (2025)
Nguyen, K., Nguyen, H., Pham, T., Ho, N.: Lightspeed geometric dataset distance via sliced optimal transport. arXiv preprint arXiv:2501.18901 (2025)
2025 arXiv
-
[26]
Journal of Machine Learning Research18(174), 1–54 (2017)
Nogueira, S., Sechidis, K., Brown, G.: On the stability of feature selection algo- rithms. Journal of Machine Learning Research18(174), 1–54 (2017)
2017
-
[27]
In: SISAP
Okkels, C.B., Thordsen, E., Aumüller, M., Zimek, A., Schubert, E.: Approximate single-linkage clustering using graph-based indexes: Mst-based approaches and in- cremental searchers. In: SISAP. pp. 233–247 (2025)
2025
-
[28]
Parsa, M.G., Zare, H., Ghatee, M.: Unsupervised feature selection based on adap- tive similarity learning and subspace clustering. Eng. Appl. Artif. Intell.95, 103855 (2020)
2020
-
[29]
Foundations and Trends in Machine Learning11(5-6), 355–607 (2019)
Peyré, G., Cuturi, M.: Computational optimal transport. Foundations and Trends in Machine Learning11(5-6), 355–607 (2019)
2019
-
[30]
In: Proceedings of The 33rd International Conference on Ma- chine Learning
Peyré, G., Cuturi, M., Solomon, J.: Gromov-wasserstein averaging of kernel and distance matrices. In: Proceedings of The 33rd International Conference on Ma- chine Learning. vol. 48, pp. 2664–2672. PMLR (2016)
2016
-
[31]
CoRRabs/2605.22973 (2026)
Rajabinasab, M., Houle, M.E., Chelly, O., Zimek, A.: Worse than random: The im- portance of a baseline for unsupervised feature selection. CoRRabs/2605.22973 (2026)
2026 arXiv
-
[32]
In: Similarity Search and Applications
Rajabinasab, M., Lautrup, A.D., Hyrup, T., Zimek, A.: A dynamic evaluation metric for feature selection. In: Similarity Search and Applications. pp. 65–72. Springer Nature Switzerland, Cham (2025)
2025
-
[33]
In: Similarity Search and Applications
Rajabinasab, M., Lautrup, A.D., Schneider-Kamp, P., Zimek, A.: Towards semi- supervised subspace learning for outlier detection in big data. In: Similarity Search and Applications. pp. 330–344 (2026)
2026
-
[34]
CoRRabs/2501.09591(2025)
Rajabinasab, M., Lautrup, A.D., Zimek, A.: Metrics for inter-dataset similarity with example applications in synthetic data and feature selection evaluation - ex- tended version. CoRRabs/2501.09591(2025)
2025 arXiv
-
[35]
CoRRabs/2605.23563(2026)
Rajabinasab,M., Nejad, A.M., Zimek,A.: MARS: magnitude-aware rankstatistics. CoRRabs/2605.23563(2026)
2026 arXiv
-
[36]
CoRRabs/2508.12776(2025)
Rajabinasab, M., Pakdaman, F., Gabbouj, M., Schneider-Kamp, P., Zimek, A.: Randomized PCA forest for outlier detection. CoRRabs/2508.12776(2025)
2025 arXiv
-
[37]
Expert Systems with Applications281 (2025)
Rajabinasab, M., Pakdaman, F., Zimek, A., Gabbouj, M.: Randomized pca forest for approximate k-nearest neighbor search. Expert Systems with Applications281 (2025)
2025
-
[38]
CoRRabs/2604.18227(2026)
Rajabinasab, M., Zimek, A.: FSEVAL: feature selection evaluation toolbox and dashboard. CoRRabs/2604.18227(2026)
2026 arXiv
-
[39]
International Journal of Computer Vision40(2), 99–121 (2000)
Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision40(2), 99–121 (2000)
2000
-
[40]
Journal of Machine Learning Research10(1), 1–41 (2009)
Van Der Maaten, L., Postma, E.O., Van Den Herik, H.J., et al.: Dimensionality reduction: A comparative review. Journal of Machine Learning Research10(1), 1–41 (2009)
2009
-
[41]
Springer, Berlin, Heidelberg (2009)
Villani, C.: Optimal Transport: Old and New. Springer, Berlin, Heidelberg (2009)
2009
-
[42]
In: ICML
Zhao, Z., Liu, H.: Spectral feature selection for supervised and unsupervised learn- ing. In: ICML. pp. 1151–1157 (2007)
2007
-
[43]
Zimek, A., Schubert, E., Kriegel, H.: A survey on unsupervised outlier detection in high-dimensional numerical data. Stat. Anal. Data Min.5(5), 363–387 (2012)
2012
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.