{"id":"0d27730a-d64e-4da9-81a2-1256574db4a7","arxiv_id":"2508.00954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FeatureCuts selects the feature cutoff after filter ranking by optimizing a weighted score of model performance and feature reduction, cutting computation time dramatically.","lead":"This paper introduces FeatureCuts, a method that ranks features with a statistical test and then automatically searches for the best number of top features to keep, using Bayesian optimization or golden-section search. The authors report large reductions in feature count and computation time on 15 datasets while keeping model accuracy about the same.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FS-score appears to be computed on the hold-out test set when selecting cutoff k, making final test scores and the brute-force 'ground truth' in Fig. 2 leaky; performance-maintenance claim is unsupported without a demonstrated train/validation/test split.","rationale":"In good faith, FeatureCuts is a plausible and simple extension of hybrid filter-wrapper selection, and the reported runtime savings are credible if the cutoff is evaluated on a validation split. The reader's weakest assumption was unimodality of the FS-score landscape, but the paper attempts to address that empirically through the brute-force comparison in Figure 2. The more fundamental, load-bearing issue is the ambiguity in how FS-score is computed: if the hold-out test set is used to select the cutoff, then both the final performance numbers and the validation of the optimizer are compromised. This concern is not an attack on the authors; it is a standard selection-bias check that the paper does not explicitly rule out. The proposed concrete test—rerunning with an explicit train/validation/test separation—would settle it. I therefore do not change the reader's CONDITIONAL verdict, but I emphasize that the condition should be a clean split, not merely an investigation of unimodality. Independent support for the method's speed and feature reduction remains plausible, but the performance-maintenance claim hinges on this split.","tokens_in":15625,"tokens_out":5197,"duration_ms":48345,"concrete_test":"Re-run the FeatureCuts pipeline with a clean split: use only training data (e.g., inner 5-fold cross-validation) to compute FS-score for each candidate cutoff k, select k*, then train the final XGBoost model and evaluate on the untouched hold-out test set. Compare these test scores and chosen cutoffs against Table IV and Figure 2. If the re-run test scores drop by more than ~0.01–0.02 on several datasets or the cutoffs shift materially, the reported performance-maintenance claim is an artifact of test-set selection. A secondary check: inspect the code (if released) for any call that evaluates FS-score on the test split during cutoff optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—FeatureCuts maintains model performance while reducing features and time—depends on the FS-score used to select the cutoff being an unbiased estimate of generalization. Section II.B defines FS-score using 'model test score after feature selection,' and Section IV.B states that brute-force search 'trained a model using just the top k features and recorded the resulting FS-score' for every k. Section III.E only says 'nested, stratified 5-fold cross-validation and hold-out test set' without specifying which split supplies the FS-score. If the hold-out test set is used to evaluate candidate cutoffs—as the repeated phrase 'test score' suggests—then the reported test scores in Tables III–VI are optimistically biased because k was selected on the same test set, and Figure 2's alignment of Golden Section Search/Bayesian Optimization with the brute-force maximum is a within-test-set fit that cannot establish that the optimizer finds a generalizing cutoff. This is more load-bearing than the unimodality concern: even a perfectly unimodal FS-score landscape would not rescue the performance-maintenance claim if the objective itself is evaluated on test data. The additional tuning of FS-score weights (ws=50, wf=1) on the evaluation datasets compounds the selection leakage, as does the absence of released code for independent audit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FeatureCuts, a hybrid feature selection method for large datasets. After ranking features by an F-value filter, it selects the optimal cutoff number k by maximizing a weighted harmonic mean (FS-score) of the feature-reduction percentage and the model score, using Bayesian Optimization (5 initial points, 5 iterations) or a discrete adaptation of Golden Section Search (10 iterations). Optionally, the selected top-k features are fed into an evolutionary wrapper method (PSO, GWO, WOA, or SCA) for further reduction. The authors evaluate on 14 public datasets and one proprietary dataset, comparing against Boruta, ReliefF, and the four wrapper methods. They report that FeatureCuts achieves on average 67% feature reduction in about one minute, versus 3–4 hours for wrappers, while maintaining competitive test scores, and that using it before PSO improves feature reduction from 57% to 82% with one-third of the runtime.","tokens_in":15898,"tokens_out":7445,"duration_ms":68188,"significance":"The problem of choosing a cutoff after filter ranking is real and underexplored, and the proposed reformulation as black-box optimization is simple and potentially practical. The paper's strengths include a wide range of test datasets (including LLM-based embedding features), a comparison to a brute-force optimum, and the demonstration of large speedups over wrapper methods. However, the evaluation protocol currently does not establish that the reported test scores are unbiased: the FS-score used for cutoff selection may be computed on the same test set that is later reported, and several hyperparameters (FS-score weights, filter metric) are tuned on the evaluation datasets themselves. These issues, if confirmed, would invalidate the central empirical claims. The unimodality assumption behind the cutoff optimizers is also unverified. Should a proper nested validation protocol confirm the results, the method would be a useful contribution to the feature selection literature.","major_comments":[{"comment":"The manuscript does not specify which data split is used to compute the model score S that enters the FS-score during cutoff selection. Section II.B defines S as 'model score after feature selection' and Section III.E mentions 'nested, stratified 5-fold cross-validation and hold-out test set,' but it never states whether S is computed on the hold-out test set, on a validation fold, or on training data. If S is the same hold-out test score reported in Tables IV and VI, then the cutoff k is selected on the test set and the reported test scores are optimistically biased; the brute-force 'ground truth' optimum in Section IV.B would likewise be a test-set maximum, so Figure 2's alignment between the optimizers and the brute-force optimum would not demonstrate generalization. This issue is load-bearing for the central claim that FeatureCuts 'maintains model performance' while reducing features. The authors must clarify the exact data flow and, if the current protocol uses the test set for cutoff selection, repeat the experiments with a separate validation set for model selection and a fresh hold-out set for final reporting.","section":"Sections III.E and II.B"},{"comment":"The FS-score weights (ws=50, wf=1) and the choice of F-value as the ranking metric are selected after observing results on the same 14 datasets that are later used to report the final evaluation. Section II.B states that these weights 'achieved a good balance between feature reduction and model performance on our evaluation datasets,' and Section IV.A selects the filter metric by comparing performance 'across each evaluation dataset.' This introduces selection bias: the reported averages in Tables III–VII are not independent of the hyperparameter choices, so the claimed cross-dataset generalizability is overstated. The paper should either reserve a portion of the data (or a separate set of datasets) for hyperparameter selection, use a nested cross-validation procedure that treats hyperparameters as part of the model, or at least acknowledge that the results are conditional on the chosen weights and metric.","section":"Sections II.B and IV.A"},{"comment":"The optimization methods assume that the FS-score as a function of k is sufficiently smooth and (for Golden Section Search) unimodal, but the paper does not test this assumption. The authors state that the automated cutoffs 'typically align closely' with the brute-force maximum (Section IV.B), but no quantitative metric (e.g., distance between selected k and k*, or FS-score gap) is reported, and no analysis of the number of local maxima in the FS-score curves is given. Without such evidence, the convergence and robustness of the 10-iteration Golden Section Search and 5-iteration Bayesian Optimization to the global optimum are not established, which is important because a suboptimal cutoff would weaken the feature-reduction and time-saving claims. The authors should report quantitative alignment with the brute-force optimum for all datasets and examine the landscape for multimodality, or alternatively use a multi-start heuristic and compare results.","section":"Section II.C"},{"comment":"The claim that FeatureCuts 'maintains model performance' is supported only by point estimates of the average test score; no confidence intervals, per-dataset paired comparisons, or significance tests are provided. For example, Table III reports an average test score of 0.817 ± 0.17 for FC GS versus 0.825 ± 0.18 for Boruta, and Table V shows FC GS→PSO at 0.814 ± 0.18 versus PSO at 0.819 ± 0.18. Given the large standard deviations, these differences are not distinguishable from noise, and the conclusion that performance is 'maintained' (as opposed to degraded or improved) is not statistically supported. The authors should report per-dataset performance differences and a paired significance test (e.g., Wilcoxon signed-rank test) to support the performance-maintenance claim.","section":"Sections V.A and V.B"}],"minor_comments":[{"comment":"In the 'bss' row, the number of 'Used Samples' (20,000) exceeds the number of instances in the dataset (13,991); please verify the sampling procedure or correct the table.","section":"Table I"},{"comment":"The legend lists a dataset 'rcv1_one_col_target,' but Table I and the text refer to this dataset as 'relathe'; the naming should be made consistent.","section":"Figures 3 and 4"},{"comment":"The header contains the typo 'REUSLTS' instead of 'RESULTS.'","section":"Table V"},{"comment":"The evaluation is described as using 'nested, stratified 5-fold cross-validation and hold-out test set,' but the number of outer folds and the construction of the hold-out set are not described; please provide details for reproducibility.","section":"Section III.E"},{"comment":"The paper does not include a link to the implementation or a description of the experimental environment (hardware, library versions); the computation-time comparisons would benefit from such information.","section":"General"},{"comment":"The text refers to 'F-value (f-classif and f-regres),' but Table II uses the label 'F classif'; please unify the notation.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from an explicit data and code availability statement; the proprietary dataset is a minor concern, but the evaluation ambiguity described in major comment 1 is central. I would encourage the editor to require the authors to specify the exact code and data pipeline before a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is simple: after an F-value ranking, treat the number of kept features k as a one-dimensional black-box function of an FS-score (weighted harmonic mean of reduction and model score) and optimize it with Bayesian Optimization or Golden Section Search. That framing doesn't appear in the cited literature, and it is a reasonable, cheap thing to try. The paper does a solid job on breadth: 14 public datasets plus an industry set, both classification and regression, and LLM-embedding features. The reported time savings are large and plausible given the 10-iteration GSS budget. The hybrid results with PSO also show a real effect (82% reduction vs 57%, time down from 3h to 1h) with roughly stable test scores.\n\nThe soft spots are more serious than the paper lets on. First, the FS-score is defined using \"model test score after feature selection\" (Section II.B), and the paper never says which split that score comes from. If the hold-out test set is used to pick k, then the final \"test score\" in Tables III–VI is a test-set fit, and the Figure 2 alignment with the brute-force optimum is within-sample, not evidence of generalization. This is load-bearing: the central claim is \"maintains model performance,\" and that claim only holds if the cutoff is chosen without touching the test labels. The fix is easy—use the validation fold for FS-score, report test once—but the paper must state it explicitly.\n\nSecond, the FS-score weights (50:1) and the choice of F-value over MI are both tuned by looking at the same 14 datasets used for the final evaluation. That's selection bias; it doesn't kill the idea, but it means the averages in Table III are optimistic.\n\nThird, GSS assumes unimodality. Figure 2 shows noisy, sometimes multi-peaked FS-score curves (e.g., the airline and madelon panels), so 10 iterations of GSS can plausibly land off the global optimum. The paper should present a unimodality diagnostic or at least a sensitivity analysis.\n\nFourth, and this is a citation-integrity flag: reference [27], for the Sine Cosine Algorithm, is attributed to \"J. Smith and A. Doe\" in the Journal of Computational Intelligence, vol. 35, no. 4, pp. 1234–1245, 2021. That looks like a placeholder, not a real paper; the actual SCA reference is Mirjalili, Knowledge-Based Systems 2016. If a reviewer spots this, it will damage the paper's credibility.\n\nBottom line: the idea is worth taking seriously, the experiments are broad, and the flaws are fixable. The evaluation split issue is the one that needs a definitive answer before the performance-maintenance claim can be trusted. This deserves a proper peer review, but it should come back with major revisions, not a quick accept.\n\nRecommendation: send it to review; require the train/validation/test separation, weight ablation, and correction of the suspect reference.","headline":"Practical cutoff-optimization idea with credible speedups, but the evaluation split is ambiguous and one reference looks fabricated — needs careful revision before the numbers can be trusted.","tokens_in":16425,"tokens_out":3019,"would_cite":false,"duration_ms":28456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the optimal filter cutoff can be found automatically in about a minute with Bayesian Optimization or Golden Section Search, matching wrapper methods on performance while cutting far more features.","keywords":["feature selection","filter ranking","cutoff optimization","Bayesian optimization","golden section search","F-value ranking","hybrid feature selection","LLM embeddings"],"falsifier":"Take a synthetic dataset with 1,000 features where features 1–30 are strong predictors, features 400–450 are also strong predictors of a separate class pattern, and features 31–399 are noise, then run FeatureCuts (F-value ranking, GSS with 10 iterations and BO with 5 initial points and 5 iterations) and compare the selected $k^*$ to the brute-force argmax of FS-score over all $k$. If the optimizer reports a cutoff near 30 while the global FS-score peaks near 420, the unimodality assumption fails and the claimed near-optimal approximation breaks.","tokens_in":15390,"feed_emoji":"✂️","tokens_out":5493,"duration_ms":51441,"temperature":0.7,"pith_summary":"This paper claims that the hard part of hybrid feature selection—deciding how many top-ranked features to keep—can be solved automatically by treating the cutoff as a black-box optimization problem. After ranking features with an F-test, FeatureCuts runs Bayesian Optimization or Golden Section Search over the cutoff $k$, scoring each candidate with the FS-score, a weighted harmonic mean of feature reduction and model performance. Across 14 public datasets and one industry dataset, the cutoff found this way averages 67 percent feature reduction in about one minute of compute, versus 19 minutes for Boruta and 3–4 hours for evolutionary wrappers, with test scores essentially unchanged. The paper also shows that feeding the resulting top-$k$ features into Particle Swarm Optimization gives 82.6 percent reduction and 66 percent less time than PSO alone. The insight matters because fixed cutoffs or exhaustive searches over $k$ do not scale to the wide, high-dimensional datasets now common with LLM embeddings.","feed_headline":"One minute finds the best feature cutoff","feed_subtitle":"Optimizing the cutoff after F-value ranking cuts the most features while keeping model scores high.","key_machinery":"The FS-score, a weighted harmonic mean of feature reduction and model performance, is the objective that carries the argument. It turns the cutoff decision into a one-dimensional black-box maximization over $k \\in \\{1,\\dots,N\\}$, so the paper can apply Bayesian Optimization and Golden Section Search with very few model trainings. The golden ratio conjugate $\\varphi = (\\sqrt{5}-1)/2$ sets the bracket widths in the Golden Section Search, and the adaptation to discrete integers picks $k^* = \\arg\\max_{k \\in \\{\\lfloor a\\rfloor, \\lceil b\\rceil\\}} FSS(k)$ after ten iterations.","core_discovery":"The central discovery is that the optimal filter cutoff $k^* = \\arg\\max_k FSS(k)$ can be approximated with a handful of model evaluations, not thousands. The authors define the FS-score $FSS(k) = \\frac{w_s + w_f}{w_s/S + w_f/(1 - F_r/F_b)}$ with weights 50 on model score and 1 on feature reduction, and show empirically that this score as a function of $k$ behaves like a smooth, nearly unimodal curve across their datasets. Both Bayesian Optimization (5 initial points, 5 iterations) and Golden Section Search (10 iterations, adapted to integer intervals) land close to the brute-force maximum FS-score on the validation plots, and the resulting feature subsets match or beat wrapper methods on feature reduction while keeping test scores within about 0.01 of the no-selection baseline. This is an extension claim: the cutoff-selection step that previous hybrid methods settled with fixed percentages or arbitrary feature counts can itself be optimized with negligible overhead.","pith_inferences":["If the FS-score curvature is reliably unimodal, the same one-dimensional optimizer could be reused to tune other threshold decisions, such as variance thresholds or correlation cutoffs, with the same near-zero overhead.","The unimodality assumption could be probed directly: on datasets with two well-separated blocks of informative features, the FS-score may become bimodal, and FeatureCuts would then need more than ten GSS iterations or a multi-start variant.","The fixed weight ratio of 50:1 biases the cutoff toward performance; a user who values interpretability or deployment cost more could re-run the search with a higher feature-reduction weight and get the same speed advantage.","A natural extension is to couple the cutoff search with a learning curve so that the model score at cutoff $k$ is estimated from a subsample, slashing the per-evaluation cost further on very large data."],"forward_implications":["FeatureCuts alone can be used as a drop-in replacement for fixed-cutoff filter selection, giving about 67% average feature reduction in about a minute on datasets up to 5,000 features.","As the filtering stage before PSO, it lifts feature reduction from 57% to 83% and cuts runtime from roughly 3 hours to about 1 hour while the test score moves only from 0.819 to 0.814.","The method carries over to regression tasks, which use F-statistic ranking, and to features derived from LLM embeddings, where it matched the best test scores on the text datasets in the paper.","The same optimization shell can host any filter metric and any wrapper algorithm, not just F-value and the four evolutionary methods tested."],"supporting_citations":[{"why":"Supplies the Bayesian Optimization routine used to search over the cutoff with 5 initial points and 5 iterations.","marker":"[21]"},{"why":"Provides the Golden Section Search implementation that the paper adapts to discrete integer cutoffs with the golden ratio conjugate.","marker":"[22]"},{"why":"Defines the Particle Swarm Optimization wrapper used both as a baseline and as the downstream wrapper in the hybrid experiments.","marker":"[24]"},{"why":"Supplies Boruta, a state-of-the-art baseline that FeatureCuts is compared against for feature reduction, runtime, and model score.","marker":"[43]"},{"why":"Supplies ReliefF, the interaction-aware ranking baseline that FeatureCuts is compared against when combined with Golden Section Search.","marker":"[44]"},{"why":"Motivates hybrid filter-based evolutionary feature selection and illustrates the fixed-cutoff practice that FeatureCuts replaces.","marker":"[13]"},{"why":"Provides another hybrid filter-genetic baseline using a fixed top-5% cutoff, which FeatureCuts improves upon.","marker":"[14]"},{"why":"Provides the evolutionary wrapper implementations (PSO, GWO, WOA, SCA) used in the comparisons and hybrid experiments.","marker":"[23]"},{"why":"Supplies the public datasets that form the evaluation benchmark for the method.","marker":"[28]"}],"fun_headline_variants":["FeatureCuts: Find the best feature cutoff in minutes, not hours","Optimize the feature cutoff: FeatureCuts does it in a few model runs","FeatureCuts: 15% more feature reduction, 99.6% less computation","One-minute feature cutoff optimization boosts ML efficiency","FeatureCuts: The smart way to pick how many features to keep"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speed advantage rests on the FS-score being a smooth, nearly unimodal function of the cutoff $k$, so that ten Golden Section iterations or five Bayesian Optimization iterations find the same peak a brute-force search would; if the score has two distant local maxima of similar height, the optimizer can stop at the wrong cutoff.","fun_headline_variants_meta":{"raw":{"variants":["FeatureCuts: Find the best feature cutoff in minutes, not hours","Optimize the feature cutoff: FeatureCuts does it in a few model runs","FeatureCuts: 15% more feature reduction, 99.6% less computation","One-minute feature cutoff optimization boosts ML efficiency","FeatureCuts: The smart way to pick how many features to keep"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2976,"prompt_tokens":891,"completion_tokens":2085,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1989}},"tokens_in":507,"tokens_out":2085,"duration_ms":14250,"temperature":1.0,"reasoning_tokens":1989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:14:18.406070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic dataset with 1,000 features where features 1–30 are strong predictors, features 400–450 are also strong predictors of a separate class pattern, and features 31–399 are noise, then run FeatureCuts (F-value ranking, GSS with 10 iterations and BO with 5 initial points and 5 iterations) and compare the selected $k^*$ to the brute-force argmax of FS-score over all $k$. If the optimizer reports a cutoff near 30 while the global FS-score peaks near 420, the unimodality assumption fails and the claimed near-optimal approximation breaks.","supporting_citations":[{"cited_title":"Golden-section search,","cited_arxiv_id":null,"evidence_quote":"Provides the Golden Section Search implementation that the paper adapts to discrete integer cutoffs with the golden ratio conjugate."},{"cited_title":"Binary genetic swarm optimization: A combination of ga and pso for feature selection,","cited_arxiv_id":null,"evidence_quote":"Defines the Particle Swarm Optimization wrapper used both as a baseline and as the downstream wrapper in the hybrid experiments."},{"cited_title":"Feature selection with the boruta package,","cited_arxiv_id":null,"evidence_quote":"Supplies Boruta, a state-of-the-art baseline that FeatureCuts is compared against for feature reduction, runtime, and model score."},{"cited_title":"Benchmarking Relief-Based Feature Selection Methods for Bioinformatics Data Mining","cited_arxiv_id":"1711.08477","evidence_quote":"Supplies ReliefF, the interaction-aware ranking baseline that FeatureCuts is compared against when combined with Golden Section Search."},{"cited_title":"Feature selec- tion in high dimensional data by a filter-based genetic algorithm,","cited_arxiv_id":null,"evidence_quote":"Motivates hybrid filter-based evolutionary feature selection and illustrates the fixed-cutoff practice that FeatureCuts replaces."},{"cited_title":"Hybrid filter and genetic algorithm-based feature selection for improving cancer classification in high-dimensional microarray data,","cited_arxiv_id":null,"evidence_quote":"Provides another hybrid filter-genetic baseline using a fixed top-5% cutoff, which FeatureCuts improves upon."},{"cited_title":"Py fs: A python package for feature selection using meta-heuristic optimization algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary wrapper implementations (PSO, GWO, WOA, SCA) used in the comparisons and hybrid experiments."},{"cited_title":"The uci machine learning repository,","cited_arxiv_id":null,"evidence_quote":"Supplies the public datasets that form the evaluation benchmark for the method."}],"review_version":1}