REVIEW 5 major objections 7 minor 28 references
An Empirical Study of Feature Selection Granularity
T0 review · 5 major / 7 minor · reviewed 2026-07-31 · grok-4.5
Pith's one-line read Greedy recursive elimination almost always beats one-shot global ranking for the same feature-selection criterion.
desk verdict Solid paired experiment on selection granularity with a real supervised win, but the multi-metric “almost consistent” claim does not match several of their own rank diagrams. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Greedy recursive elimination: at each step the current least-important feature is discarded and the base importance estimator is re-run on the remaining features, producing an elimination order that becomes the final ranking.
What would settle it
Re-run the identical head-to-head comparison on several dense or sparse datasets with thousands of features; if the recursive variants no longer improve accuracy, AUC, clustering quality or angle-difference scores relative to one-shot ranking, the central claim fails.
Extended reading notes
Core claim
When the same feature-importance criterion is applied either once globally or iteratively by removing the least important feature and re-scoring, the iterative design yields higher-quality selected subsets on nearly every evaluation metric examined. The discrepancy is largest among the top-ranked features and shrinks only as larger fractions of the feature set are retained.
Load-bearing premise
The finding is assumed to carry over from the moderately dimensional datasets used (at most a few hundred features) to the truly high-dimensional regimes where the curse-of-dimensionality argument is usually invoked.
Editorial extensions
If this is right
- Practitioners can obtain better feature subsets from existing selectors simply by wrapping them in recursive elimination, without inventing new criteria.
- One-shot global rankings should be treated as approximate when many noisy or collinear features are present.
- The computational overhead of recursion becomes the main practical barrier, so batch or subspace variants become natural next engineering steps.
- Any new feature-selection method whose scores depend on the current feature set should be evaluated under both one-shot and recursive protocols.
Reading between the lines
- The same masking effect may also distort unsupervised filters and wrapper methods that were excluded from the study, suggesting a broader design principle.
- If removing several least-important features per round still preserves most of the gain, recursive selection could become practical for much higher dimensions.
- The result supplies a concrete reason why feature-selection stability metrics often disagree across algorithms: they may be measuring different points on a continuum of granularity rather than purely different criteria.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks an algorithmic-design question about feature selection: does computing importance scores once over the full feature set (global/one-shot) differ from greedily removing the least important feature and re-evaluating importance at each step (iterative/RFE-style)? The authors implement five supervised selectors (Random Forest MDI, XGBoost MDI, ReliefF, LASSO, permutation importance) in both variants, evaluate them on 28 PMLB datasets across nested subset budgets (5%–100% of features) with 5-fold CV, and compare them on ACC, AUC, clustering accuracy (CLSACC), NMI, a PCA-based model-agnostic metric (AAD), ranking similarity, and runtime, using both standard average ranks and the authors' magnitude-aware rank statistic (MARS). The manuscript claims the iterative design improves feature-selection quality 'almost consistently' across metrics, at higher computational cost, and interprets this as evidence that high dimensionality also degrades the importance-estimation step itself.
Significance. The question is genuinely useful and, to my knowledge, not systematically studied: practitioners routinely choose between one-shot global importance ranking and recursive elimination, and paired evidence across five diverse selectors (tree MDI, boosting, ReliefF, LASSO, permutation) on 28 PMLB datasets with 5-fold CV is a real contribution. The supervised result — iterative variants occupying the top rank positions on ACC (Fig. 3) — is, if it holds up under paired testing, actionable guidance. The honest negative/mixed results on CLSACC and AAD are also valuable and would strengthen the paper if reported as such rather than papered over. However, the current version overclaims a cross-metric 'almost consistent' improvement that its own figures refute, and the dimensional regime studied (≤240 features) does not match the curse-of-dimensionality motivation. The contribution is salvageable and worthwhile at a re-scoped claim; it is not currently publishable as stated.
major comments (5)
- [§5.2, Figs. 4–7 captions] The captions of Figs. 4, 5, and 7 each state that 'Both MARS and Standard rank statistics agree on the superiority of the iterative approach,' but the displayed rankings contradict this. Fig. 4 (AUC), MARS diagram: standard XGB ranks first (9.24) and iterative XGB last (453.53); standard Permutation and ReliefF also outrank their iterative variants. Fig. 5 (CLSACC), standard-rank diagram: every standard selector ranks ahead of its iterative counterpart (Lasso 4.04 < Lasso_iter 4.36; XGB 4.84 < XGB_iter 5.32; RF 5.39 < RF_iter 5.45; ReliefF 5.75 < ReliefF_iter 6.18; Perm 6.25 < Perm_iter 7.43). Fig. 7 (AAD), both diagrams predominantly favor the standard variants of ReliefF, RF, Perm, and XGB. The body text for CLSACC is more careful ('not as consistently superior as for the supervised measure'), but the captions and the Abstract's 'almost consistently' are not supported by the paper's ow
- [Abstract; §5.2; §6] Related to the above: the Abstract and §6 claim the greedy design 'improves the overall feature selection quality almost consistently,' yet by my reading of the displayed rank diagrams the iterative variant is clearly superior only for ACC (Fig. 3, where the top three positions are iterative in both statistics) and for AUC under standard ranks. NMI (Fig. 6) is mixed, and CLSACC/AAD point the other way under standard ranks. Since the paper's contribution is precisely the empirical claim about granularity, the mismatch between claim and evidence is load-bearing. A revision should either (a) restrict the claim to supervised downstream metrics and explain why supervised selectors improving supervised metrics is itself the interesting (and perhaps expected) result, or (b) provide additional analysis supporting the broader claim.
- [§5.2 (rank analysis methodology)] The aggregate critical-difference diagrams pool all ten variants (five base selectors x two designs). This shows overall placement but never directly tests the paired comparison the paper is actually about: RF vs RF_iter, XGB vs XGB_iter, etc., across the 28 datasets. Standard practice (Demšar, ref. [8], which the paper cites) would be a Wilcoxon signed-rank test per base-estimator pair per metric. Without paired tests, statements like 'the iterative variants occupy superior rank positions' do not establish that making any given selector iterative significantly helps; a single strong base method can shift pooled ranks. This is fixable with the existing experimental output.
- [§4.2; §6] The motivating story is the curse of dimensionality obscuring feature-importance estimation, yet the benchmark deliberately caps dimensionality at 240 features (§4.2), with most datasets below 100. The only genuinely high-dimensional example, COIL-20 (1024 features, Fig. 1), is used as an anecdotal t-SNE illustration and is excluded from the quantitative benchmark. The exclusion rationale in §4.2 (sparse high-d data makes granular comparison difficult) is reasonable, but then the conclusion in §6 that high dimensionality degrades feature selection analogously to distance concentration is asserted, not demonstrated — the experiments show an iterative-vs-global gap in low-to-moderate dimensions, which if anything undercuts the dimensionality narrative. Either add experiments in a higher-dimensional regime or reframe the contribution as a granularity study independent of the curse-of-dimens
- [§1 vs §5 and Table 1] The dataset count is internally inconsistent: §1 states 'extensive experiments across 38 datasets,' §5 states '28 datasets,' and Table 1 lists exactly 28. If 10 datasets were dropped, the selection criterion must be stated; if 38 is simply wrong, correct it. As written, a reader cannot determine whether there is selection on results.
minor comments (7)
- [§4 vs §5] CLSACC is defined in §4 as 'Clustering Accuracy' but listed in the §5 introductory paragraph as 'Class-Weighted Accuracy.' These are different metrics; unify the terminology.
- [§3, Eq. (3)] Eq. (3) and surrounding text: S_0 = {1,...,n} uses n (the number of instances) for the feature index set; it should be d. Also the elimination order is stored in 'a sequence ε' but features are then described as 'appearing late in π' — ε and π are used inconsistently.
- [title page; Abstract; §3; §5] Typos: affiliation 'Mahematics'; §3 'might be effected' (affected); Abstract 'on the expense' (at the expense); §5.1 'two show if there is a performance difference' (to show); §5.3 'most of the times, a one-time step' is garbled.
- [Fig. 1] Fig. 1's t-SNE comparison is suggestive but anecdotal: t-SNE projections are stochastic and the visual 'better discrimination' is not quantified. Please label it explicitly as a motivating illustration, and note that COIL-20 does not appear in the benchmark (Table 1).
- [§5.2; ref. [24]] MARS (ref. 24) is a CoRR preprint by the same authors and is one of the two rank statistics used to support the central claim. This should be disclosed in the text where MARS is introduced, and the robustness of conclusions to using only standard ranks (ref. [8]) should be stated explicitly — particularly since for AUC the two statistics disagree.
- [§1] The statement in §2 that there is an 'absence of prior studies investigating the granularity of the feature selection process' is strong given that RFE (refs. [9, 12]) and stepwise/wrapper methods (ref. [13]) are established; the contribution is better framed as a systematic cross-method, cross-metric comparison of the two granularities rather than the first study of iterative elimination.
- [§4.1] Hyperparameter choices (50 estimators for RF/XGB but only 20 for the Permutation backbone; LASSO C=0.5 fixed across all datasets) are stated but not justified or sensitivity-checked. Since the Permutation variant uses a weaker backbone than RF, the RF vs Permutation cross-comparisons in the pooled diagrams are not on equal footing; a brief note suffices.
Circularity Check
No significant circularity: the paper is an empirical head-to-head of two algorithmic designs on external metrics and datasets, not a derivation that forces its conclusion.
full rationale
The central claim is that greedy recursive elimination yields better feature-selection quality than one-shot global ranking of the same criterion. That claim is tested, not derived: five base selectors are run under both designs on PMLB datasets, and performance is reported primarily via standard external metrics (ACC, AUC, CLSACC, NMI) under 5-fold CV. Nothing in the methodology algebraically entails that the iterative variants must win; the ranking similarity heatmaps even document that the two designs select different features. Recursive elimination itself is a known wrapper pattern (Guyon et al., Kohavi & John), not an author-private uniqueness result. Self-citations to FSEVAL, AAD, FSDEM, and MARS supply the benchmarking harness and secondary aggregation views; they do not define the target quantities being compared and do not make the supervised gains true by construction. Overstatement of multi-metric consistency relative to some rank diagrams is a correctness/overclaim issue, not circularity. Score 0 is therefore the proportionate finding.
Assumptions & free parameters
free parameters (3)
- RF/XGB n_estimators=50; Permutation backbone 20 trees, 3 shuffles; ReliefF k=10; LASSO C=0.5 OvR =
as listed in §4.1
- Subset budgets {5%,10%,...,100%} and top-k similarity cutoffs {5..25%} =
5% steps; similarity at 5–25%
- Maximum dataset dimensionality cutoff (~240 features) =
max d=240 in Table 1
assumptions (5)
- domain assumption Feature importance from the chosen estimators remains meaningful after each single-feature deletion and can be recompared across nested subsets.
- domain assumption PMLB classification datasets with 5-fold CV and downstream ACC/AUC/clustering/AAD are adequate proxies for feature-selection quality.
- ad hoc to paper Distance-based selectors should be excluded because distance concentration would confound the granularity comparison.
- ad hoc to paper Sparse ultra-high-dimensional regimes are unsuitable for granular comparison because many subsets can look equally good.
- domain assumption Standard rank and MARS aggregate ranks over datasets are valid for declaring method superiority.
Cite this review
Pith. "Pith review of An Empirical Study of Feature Selection Granularity." pith.science (2026). https://pith.science/paper/O5YSJ363
@misc{pith2026260724145,
author = {Pith},
title = {Pith review of: An Empirical Study of Feature Selection Granularity},
year = {2026},
howpublished = {\url{https://pith.science/paper/O5YSJ363}},
note = {Machine review of arXiv:2607.24145}
}
read the original abstract
Feature selection aims to identify the most informative and relevant features for a given dataset, either in terms of capturing the underlying data structure and distribution better, or with respect to the performance on a downstream task. Existing research in this area has largely focused on developing novel algorithms (in both supervised and unsupervised settings), proposing new evaluation metrics and frameworks, or benchmarking the performance of existing methods. In this work, we examine feature selection through an algorithmic design perspective. Conventional feature selection algorithms typically compute feature importance scores globally across the entire feature set and then select the top-ranked features in a single step. However, this approach raises a critical question: Can the presence of less informative (or noisy) features mask or obscure the true importance of other, more relevant features? In other words, would a recursive strategy, where features are removed one by one while re-evaluating importance at each step, yield different and potentially better results than the standard global ranking approach? To answer this question, we conduct an extensive empirical study using five diverse feature selection algorithms. We implement each algorithm under both the conventional global selection design and the greedy recursive elimination design. We then analyze the impact of this algorithmic choice, both individually for each method and collectively across all methods, on a range of standard feature selection evaluation metrics. The empirical evaluation results show that the greedy approach improves the overall feature selection quality almost consistently, albeit on the expense of higher computational cost, supporting our initial expectation that the curse of dimensionality also obscures the ways of mitigating it.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[8]
JMLR7, 1–30 (2006)
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. JMLR7, 1–30 (2006)
2006
-
[1]
Bioinformatics26(10), 1340–1347 (2010)
Altmann, A., Toloşi, L., Sander, O., Lengauer, T.: Permutation importance: a corrected feature importance measure. Bioinformatics26(10), 1340–1347 (2010)
2010
-
[2]
IEEE Trans
Amsaleg, L., Bailey, J., Barbe, A., Erfani, S.M., Furon, T., Houle, M.E., Radovanovic, M., Nguyen, X.V.: High intrinsic dimensionality facilitates adver- sarial attack: Theoretical evidence. IEEE Trans. Inf. Forensics Secur.16(2021)
2021
-
[3]
Anderberg, A., Bailey, J., Campello, R.J.G.B., Houle, M.E., Marques, H.O., Radovanovic, M., Zimek, A.: Dimensionality-aware outlier detection. In: SDM. pp. 652–660. SIAM (2024)
2024
-
[4]
Journal of AI and Data Mining9(1), 31–38 (2021).https: //doi.org/10.22044/jadm.2020.9376.2072
Asadi Amiri, S., Rajabinasab, M.: Face recognition using color and edge orientation difference histogram. Journal of AI and Data Mining9(1), 31–38 (2021).https: //doi.org/10.22044/jadm.2020.9376.2072
arXiv 2021
-
[5]
nearest neighbor
Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is “nearest neighbor” meaningful? pp. 217–235 (1999)
1999
-
[6]
Machine learning45, 5–32 (2001)
Breiman, L.: Random forests. Machine learning45, 5–32 (2001)
2001
-
[7]
In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. pp. 785–794 (2016)
2016
Show all 28 references
-
[9]
In: 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, FL, USA, December 17-20, 2018
Escanilla, N.S., Hellerstein, L., Kleiman, R., Kuang, Z., Shull, J.D., Page, D.: Recursive feature elimination by sensitivity testing. In: 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, FL, USA, December 17-20, 2018. pp. 40–47. IE...
2018
-
[10]
IEEE Trans
François, D., Wertz, V., Verleysen, M.: The concentration of fractional distances. IEEE Trans. Knowl. Data Eng.19(7), 873–886 (2007)
2007
-
[11]
Journal of machine learning research3(Mar), 1157–1182 (2003)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of machine learning research3(Mar), 1157–1182 (2003)
2003
-
[12]
Machine learning46, 389–422 (2002) 16 M
Guyon, I., Weston, J., Barnhill, S., Vapnik, V.: Gene selection for cancer classifi- cation using support vector machines. Machine learning46, 389–422 (2002) 16 M. Rajabinasab & A. Zimek
2002
-
[13]
Artificial In- telligence97(1-2), 273–324 (1997).https://doi.org/10.1016/S0004-3702(97) 00043-X,https://doi.org/10.1016/S0004-3702(97)00043-X
Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artificial In- telligence97(1-2), 273–324 (1997).https://doi.org/10.1016/S0004-3702(97) 00043-X,https://doi.org/10.1016/S0004-3702(97)00043-X
1997 doi
-
[14]
In: Euro- pean conference on machine learning
Kononenko, I.: Estimating attributes: Analysis and extensions of relief. In: Euro- pean conference on machine learning. pp. 171–182. Springer (1994)
1994
-
[15]
Phys- ical review E69(6), 066138 (2004)
Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Phys- ical review E69(6), 066138 (2004)
2004
-
[16]
ACM computing surveys (CSUR)50(6), 1–45 (2017)
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., Liu, H.: Feature selection: A data perspective. ACM computing surveys (CSUR)50(6), 1–45 (2017)
2017
-
[17]
Journal of Machine Learning Research9(86), 2579–2605 (2008),https://www.jmlr.org/papers/v9/ vandermaaten08a.html
van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research9(86), 2579–2605 (2008),https://www.jmlr.org/papers/v9/ vandermaaten08a.html
2008
-
[18]
Nene, S.A., Nayar, S.K., Murase, H.: Columbia object image library (coil-20). Tech. Rep. CUCS-005-96, Columbia University (1996)
1996
-
[19]
In: SISAP
Okkels, C.B., Thordsen, E., Aumüller, M., Zimek, A., Schubert, E.: Approximate single-linkage clustering using graph-based indexes: Mst-based approaches and in- cremental searchers. In: SISAP. pp. 233–247. Lecture Notes in Computer Science
-
[20]
BioData Mining10(36), 1–13 (Dec 2017)
Olson, R.S., La Cava, W., Orzechowski, P., Urbanowicz, R.J., Moore, J.H.: Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining10(36), 1–13 (Dec 2017)
2017
-
[21]
In: Pro- ceedings of the 2025 SIAM International Conference on Data Mining (SDM)
Rajabinasab, M., Lautrup, A., Zimek, A.: Metrics for inter-dataset similarity with example applications in synthetic data and feature selection evaluation. In: Pro- ceedings of the 2025 SIAM International Conference on Data Mining (SDM). pp. 527–537.https://doi.org/10.1137/1.9...
2025 doi
-
[22]
In: Similarity Search and Applications
Rajabinasab, M., Lautrup, A.D., Schneider-Kamp, P., Zimek, A.: Towards semi- supervised subspace learning for outlier detection in big data. In: Similarity Search and Applications. pp. 330–344. Springer Nature Switzerland, Cham (2026)
2026
-
[23]
In: Similarity Search and Applications - 17th In- ternational Conference, SISAP 2024, Providence, RI, USA, November 4-6, 2024, Proceedings
Rajabinasab, M., Lautrup, A.D., Hyrup, T., Zimek, A.: A dynamic evaluation metric for feature selection. In: Similarity Search and Applications - 17th In- ternational Conference, SISAP 2024, Providence, RI, USA, November 4-6, 2024, Proceedings. Lecture Notes in Computer Scienc...
2024 doi
-
[24]
CoRRabs/2605.23563(2026).https://doi.org/10.48550/ARXIV.2605.23563, https://doi.org/10.48550/arXiv.2605.23563
Rajabinasab,M., Nejad, A.M., Zimek,A.: MARS: magnitude-aware rankstatistics. CoRRabs/2605.23563(2026).https://doi.org/10.48550/ARXIV.2605.23563, https://doi.org/10.48550/arXiv.2605.23563
-
[25]
CoRR abs/2508.12776(2025).https://doi.org/10.48550/ARXIV.2508.12776, https://doi.org/10.48550/arXiv.2508.12776
Rajabinasab, M., Pakdaman, F., Gabbouj, M., Schneider-Kamp, P., Zimek, A.: Randomized PCA forest for unsupervised outlier detection. CoRR abs/2508.12776(2025).https://doi.org/10.48550/ARXIV.2508.12776, https://doi.org/10.48550/arXiv.2508.12776
-
[26]
Expert Systems with Applications 281, 126254 (2025).https://doi.org/https://doi.org/10.1016/j.eswa.2024
Rajabinasab, M., Pakdaman, F., Zimek, A., Gabbouj, M.: Randomized pca for- est for approximate k-nearest neighbor search. Expert Systems with Applications 281, 126254 (2025).https://doi.org/https://doi.org/10.1016/j.eswa.2024. 126254
2025 doi
- [27]
-
[28]
Journal of the Royal Statistical Society: Series B (Methodological)58(1), 267–288 (1996)
Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological)58(1), 267–288 (1996)
1996
Reviewed July 31, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.