{"id":"67253520-8203-4ad2-aa1c-662b425c030f","arxiv_id":"2507.23568","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SA-FDR uses simulated annealing guided by the Fisher discriminant ratio to find compact feature subsets for logistic regression, often beating RFE and Lasso on sparsity with comparable AUC.","lead":"SA-FDR is a feature selection algorithm that combines simulated annealing with the Fisher discriminant ratio as a fast scoring function, tested on nine public classification datasets. It often chooses fewer features than RFE or Lasso with similar accuracy, but runs slower and its evaluation has methodological gaps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set standardization in Section 4 leaks test information into training; Table 1's AUC and k* values are therefore not trustworthy, undercutting the central claim of high predictive accuracy.","rationale":"The reader's weakest assumption was the FDR proxy, but I consider the test-set standardization leak more load-bearing because it directly invalidates every quantitative result in Table 1. The proxy concern is real but partly mitigated: the algorithm selects among final replicas using cross-entropy, so a poor FDR correlation would weaken the search but not necessarily destroy the final selection. The leakage, by contrast, means the reported AUCs and k* values are not honest estimates of generalization for any method. The reader did mention the leak in their rationale, but their formal weakest_assumption points elsewhere. Since the leak is fixable by rerunning with held-out scaling and the paper's core idea may still hold, the CONDITIONAL verdict remains appropriate, hence UNCHANGED.","tokens_in":8968,"tokens_out":6089,"duration_ms":62997,"concrete_test":"Rerun the complete benchmark (all 9 datasets, SA-FDR, RFE, and Lasso) with the scaler fitted only on the training portion of each CV fold and applied to validation/test, then recompute the average k* and test AUC reported in Table 1. If any dataset's SA-FDR mean test AUC changes by more than 0.005 or its mean k* changes by more than 1.0 relative to Table 1, the published numbers are contaminated by test-set leakage and the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that 'we also calculate the mean and standard deviation of each feature from the test dataset and use these values to standardise the entire dataset.' This uses test-set statistics to transform training, validation, and test sets before model fitting. As a result, all AUC values and k* estimates in Table 1 are computed under a protocol where test information is available to the training procedure. This does not merely affect one algorithm; it biases the entire comparison. Since the paper's central claim is that SA-FDR selects sparser subsets while maintaining high predictive accuracy, the quantitative evidence for 'high predictive accuracy' is currently invalid. This is a correctness issue in the evaluation pipeline, not in the SA-FDR algorithm per se, but it means the reported results cannot be used to support the claim until the pipeline is fixed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SA-FDR, a simulated-annealing wrapper for ℓ0-norm feature selection in binary logistic models. The objective minimized during annealing is the negative Fisher discriminant ratio (FDR) of selected features, computed on randomly chosen batch matrices, with the final subset chosen among replicas by logistic cross-entropy. A cross-validation routine selects k* as the smallest subset size whose validation AUC is within one standard deviation of the best, and the final model is evaluated on a held-out test set. SA-FDR is compared with RFE and Lasso on nine UCI/public datasets including one with roughly 100,000 samples. The paper reports that SA-FDR finds sparser feature subsets with competitive or better AUC.","tokens_in":9160,"tokens_out":6322,"duration_ms":61341,"significance":"SA-FDR is a plausible combination of known components, and the FDR proxy is a computationally cheap objective that could make simulated annealing practical for wrapper-style feature selection. The paper evaluates on a diverse benchmark, including a large Loan Default dataset, and directly compares with standard RFE and Lasso baselines. The central empirical claim is currently undermined by test-set leakage in the preprocessing pipeline and by the absence of uncertainty quantification. If the evaluation is corrected and the results are reproduced, the method could still be a useful contribution, but the evidence as presented is not sufficient to support the abstract's claim of consistently high predictive accuracy.","major_comments":[{"comment":"The sentence 'we also calculate the mean and standard deviation of each feature from the test dataset and use these values to standardise the entire dataset' describes using test-set statistics to transform the data before model fitting. This leaks test information into training and validation, so all AUC values and k* estimates in Table 1 and all curves in Fig. 2 are computed under a protocol in which every algorithm has access to test-set information. This invalidates the quantitative support for the claim of 'high predictive accuracy' and for the reported sparsity-accuracy trade-off. The pipeline should be rerun with a scaler fitted only on the training portion of each fold and then applied without refitting to the validation and test portions.","section":"Section 4, data preprocessing"},{"comment":"The paper reports average k* and AUC over 'up to 20' repetitions but gives no standard deviations, confidence intervals, or significance tests. Several reported differences between algorithms are very small (e.g., Cancer AUC 0.9922 versus 0.9933; Card Default AUC 0.7214 versus 0.7220; Student k* 10.05 versus 13.88 with AUC 0.9162 versus 0.9170). Without uncertainty quantification, the claims that SA-FDR 'consistently' selects more compact subsets and 'achieves the best AUC value for some of these datasets' are not quantitatively supported.","section":"Table 1"},{"comment":"The claim that FDR 'correlates well' with logistic-regression cross-entropy is supported only by one qualitative density plot for SPECTF Heart. Since the annealing optimizes -FDR for essentially all iterations and cross-entropy is used only to select among final replicas, a divergence between FDR and cross-entropy could directly degrade the quality of the selected subsets. The authors should provide a quantitative correlation analysis over sampled subsets (e.g., Spearman or Pearson correlation between FDR and cross-entropy, or between FDR and validation AUC) across multiple datasets and several values of k.","section":"Section 2.2 and Fig. 1"},{"comment":"The k* heuristic (minimum k within one standard deviation of the maximum mean validation AUC) is acceptable, but the paper does not report the full validation-AUC curves or their standard errors for the additional datasets; only the Loan Default dataset is shown in Fig. 2. Without these curves, it is difficult to assess whether the chosen k* values are stable across folds or whether the one-standard-deviation rule selects very different k* for different algorithms.","section":"Section 3.2"}],"minor_comments":[{"comment":"The value NS = 0.5 is unusual for a parameter described as the number of sweeps, since Algorithm 1 loops for NS·K swaps. The definition and rationale for this value should be clarified.","section":"Section 4, hyperparameters"},{"comment":"The cleaning of the Loan Default dataset is described only as removing missing values and highly correlated features; the exact thresholds and procedure should be specified for reproducibility.","section":"Section 4, Loan Default dataset"},{"comment":"The term 'Receiving Operator Characteristic-Area Under the Curve' should be 'Receiver Operating Characteristic-Area Under the Curve'.","section":"Section 3.2"},{"comment":"The stopping condition 'if the mean of FDR(s) has converged' is not defined; the convergence criterion should be stated concretely.","section":"Algorithm 1"},{"comment":"No statement about code availability or random seeds is provided, which limits the reproducibility of the exact numbers in Table 1.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate, clearly written wrapper method – simulated annealing over feature subsets, scored by Fisher discriminant ratio as a cheap proxy for logistic-regression cross-entropy, with a final cross-entropy check to pick among replicas. The combination is not entirely new (simulated annealing for feature selection has been around for a long time, and the paper doesn't cite that line of work), but the specific FDR-as-proxy packaging and the benchmark against RFE and Lasso on nine datasets is a reasonable extension worth looking at.\n\nThe paper does several things well. The method is described precisely, including a pseudocode algorithm. The authors are honest about runtime being the slowest of the three. The sparsity result – they get smaller k* in most datasets – is consistent across the table and is the most credible part of the paper. The idea of using FDR as a fast objective during search and then switching to cross-entropy for final selection is a sensible engineering choice.\n\nNow the soft spots, in increasing order of seriousness.\n\nFirst, there are no error bars, confidence intervals, or significance tests anywhere in Table 1. The k* values are averages over 20 repeats, but we don't know the spread, and the differences between algorithms are often small (e.g., Cancer: 3.60 vs 5.20 k*, AUC 0.9922 vs 0.9921). Without variance, 'consistently selects more compact subsets' is a descriptive observation, not a demonstrated advantage.\n\nSecond, the proxy assumption – that FDR correlates with logistic cross-entropy – is supported by a single qualitative density plot for one dataset (Fig. 1, right). That is the load-bearing premise of the whole method, and it deserves a quantitative check across datasets and k values. The paper even says 'correlates well' without measuring the correlation.\n\nThird, and most importantly, the stress-test note is correct: Section 4 says they compute mean and std of each feature from the test dataset and use them to standardise the entire dataset. That leaks test information into training. Every AUC and k* in Table 1 is computed under that protocol. This doesn't invalidate the algorithm itself, but it invalidates the reported quantitative support for 'high predictive accuracy.' The fix is straightforward – fit the scaler on train only – and the results should be re-run.\n\nAlso minor: no code or data released, which makes the missing error bars harder to excuse, and the runtime for Loan Default (5600s per fold) is large enough that 'useful in practice' needs a qualifier.\n\nOverall: the core idea is sound and the writing is clear, but the evaluation pipeline has a correctness bug. That is fixable. I'd send this to peer review – the method deserves a proper referee, and the evaluation should be redone. I probably wouldn't cite it until the corrected results are out and the proxy correlation is quantified.","headline":"A clearly explained wrapper method that pairs simulated annealing with an FDR proxy; the idea is fine, but the evaluation leaks test-set statistics and lacks error bars, so the central accuracy claim isn't yet supported.","tokens_in":59,"tokens_out":3088,"would_cite":false,"duration_ms":59592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SA-FDR treats feature selection as global combinatorial optimisation, using the Fisher discriminant ratio as a proxy, and in benchmarks returns sparser logistic models than RFE or Lasso with equal or better AUC","keywords":["feature subset selection","simulated annealing","Fisher discriminant ratio","logistic regression","ℓ0-norm selection","sparse models","combinatorial optimisation","wrapper methods"],"falsifier":"Enumerate all feature subsets of size $k$ on a small dataset, compute the FDR and the logistic cross-entropy for each, and check whether subsets with near-maximal FDR also have near-minimal cross-entropy; if the correlation is weak, the annealing search will systematically miss the best logistic subsets on that data.","tokens_in":8816,"feed_emoji":"🎯","tokens_out":9059,"duration_ms":88700,"temperature":0.7,"pith_summary":"The paper tries to establish that feature subset selection for binary logistic classification can be treated as a combinatorial optimisation problem and solved with simulated annealing, using the Fisher discriminant ratio as a cheap stand-in for model quality during the search. On eight public tabular datasets ranging from a few hundred to about 100,000 samples, the resulting algorithm, SA-FDR, selects smaller feature subsets than recursive feature elimination or Lasso while matching or slightly improving their predictive accuracy, measured by area under the ROC curve. If true, this gives practitioners a wrapper method that finds sparse, interpretable logistic models in settings where greedy methods discard useful features early. The paper also argues the same annealing framework would work with other quality metrics and classifiers, not just logistic regression and the Fisher discriminant ratio.","feed_headline":"Simulated annealing finds sparser feature sets than greedy rivals","feed_subtitle":"A Fisher-ratio-guided search finds compact logistic models on datasets up to 100,000 samples.","key_machinery":"The load-bearing object is the Fisher discriminant ratio as a proxy for logistic-regression quality. For a candidate subset the FDR is $\\mathrm{FDR} = (\\mu_1-\\mu_0)^T S_W^{-1}(\\mu_1-\\mu_0)$, with $S_W=\\Sigma_0+\\Sigma_1$, and the annealing minimises $-\\mathrm{FDR}$ using the Metropolis acceptance rule $P=\\min\\left(1,e^{\\beta[\\mathrm{FDR}(s')-\\mathrm{FDR}(s)]}\\right)$ over $R$ replicas, with inverse temperatures raised until FDR converges. The proxy lets the search skip expensive logistic fits at every proposal; cross-entropy is evaluated only on the final replicas to pick the best subset. Batch matrices computed from different data batches add noise that the authors say helps avoid overfitting and keeps replicas from collapsing to the same local maximum.","core_discovery":"The paper's central claim is that SA-FDR reliably finds the sparsest logistic model with competitive prediction accuracy: in its benchmarks it returns the smallest optimal feature count $k^*$ in most datasets and the best test AUC on several of them, with near-best AUC on the rest. The search is structured so that simulated annealing explores feature subsets of each size $k$ under the objective $-\\mathrm{FDR}$, and at the end of annealing the replica whose logistic regression has the lowest cross-entropy is selected, which corrects for mild mismatches between FDR and true logistic quality. The authors attribute the sparsity advantage to the non-greedy exploration of the subset space: combinations that a greedy elimination procedure would discard early can be retained and tested.","pith_inferences":["A direct consequence the paper does not quantify is that the FDR proxy should be most reliable when each class is roughly unimodal and elliptically distributed; on multimodal or skewed features, high-FDR subsets may not coincide with low-cross-entropy subsets, and the final replica-selection step can only correct this within the small set of annealed candidates.","The method's dependence on $R$ replicas suggests that richer sampling schemes, such as replica exchange or parallel tempering, could improve the chance of having at least one replica near the true cross-entropy optimum; the paper notes such annealing refinements as future work but does not test them.","A testable extension would be to run SA-FDR against an exact $\\ell_0$-optimal search on small datasets ($K \\lesssim 20$) to measure how often the FDR-optimal subset is also the cross-entropy-optimal subset, quantifying the proxy gap."],"forward_implications":["On tabular binary-classification problems of the size tested, users can obtain logistic models with fewer features than RFE or Lasso yield, with accuracy that is at least competitive.","Because the objective is decoupled from the classifier, the same annealing procedure can be applied to other model-quality metrics and to non-linear classifiers such as neural networks.","The runtime, although larger than the baselines, is practical and can be reduced by tuning the number of replicas and sweeps without necessarily losing solution quality.","Sparser selected subsets make the resulting logistic models easier to interpret and less prone to overfitting in high-dimensional applications.","SA-FDR's advantage over RFE grows when useful features are correlated or redundant in ways that greedy backward elimination cannot recover."],"supporting_citations":[{"why":"Supplies the simulated annealing procedure that the whole algorithm is built on.","marker":"[24]"},{"why":"Provides the Metropolis acceptance rule used to accept or reject feature-subset proposals.","marker":"[36]"},{"why":"Gives the convergence guarantee for slow cooling schedules that justifies annealing to a near-optimal distribution.","marker":"[37]"},{"why":"Define the Fisher discriminant ratio used as the objective function and quality proxy.","marker":"[30, 31]"},{"why":"Defines the logistic regression model and its cross-entropy objective, which serve as both the final selection criterion and the benchmark model.","marker":"[29]"},{"why":"Defines recursive feature elimination, the greedy wrapper baseline against which SA-FDR's sparsity is compared.","marker":"[6]"},{"why":"Introduces Lasso and its regularisation framework, the second baseline algorithm in the benchmarks.","marker":"[1, 40]"},{"why":"Supplies the baseline implementations of RFE and Lasso used in the comparison.","marker":"[41]"},{"why":"Provides the public benchmark datasets used in the experimental evaluation.","marker":"[42]"}],"fun_headline_variants":["Simulated annealing finds sparser features than greedy search","Global search trims feature sets while preserving accuracy","Fisher-ratio annealing discovers compact, accurate models","Non-greedy feature selection beats greedy rivals","Annealing explores feature space for sparser, accurate sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feature subsets scoring high on the Fisher discriminant ratio are also the subsets that give the logistic model low cross-entropy; if that correlation breaks down, the annealing search will miss the best subsets.","fun_headline_variants_meta":{"raw":{"variants":["Simulated annealing finds sparser features than greedy search","Global search trims feature sets while preserving accuracy","Fisher-ratio annealing discovers compact, accurate models","Non-greedy feature selection beats greedy rivals","Annealing explores feature space for sparser, accurate sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1112,"prompt_tokens":831,"completion_tokens":281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":207}},"tokens_in":447,"tokens_out":281,"duration_ms":3864,"temperature":1.0,"reasoning_tokens":207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:35:36.969257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Enumerate all feature subsets of size $k$ on a small dataset, compute the FDR and the logistic cross-entropy for each, and check whether subsets with near-maximal FDR also have near-minimal cross-entropy; if the correlation is weak, the annealing search will systematically miss the best logistic subsets on that data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the simulated annealing procedure that the whole algorithm is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Metropolis acceptance rule used to accept or reject feature-subset proposals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the convergence guarantee for slow cooling schedules that justifies annealing to a near-optimal distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the logistic regression model and its cross-entropy objective, which serve as both the final selection criterion and the benchmark model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines recursive feature elimination, the greedy wrapper baseline against which SA-FDR's sparsity is compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the baseline implementations of RFE and Lasso used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the public benchmark datasets used in the experimental evaluation."}],"review_version":1}