{"id":"5d7604d4-562f-470a-8681-75061ab03bc3","arxiv_id":"2507.10591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified, publicly released benchmarking framework evaluates 17 feature selection methods across 10 Android malware datasets; LASSO, RFE, and SigAPI come out most consistent, while PCA, ReliefF, and SigPID lag.","lead":"The authors built a public framework, MH-FSF, that runs 17 feature selection methods on 10 public Android malware datasets and reports how each one changes classifier performance. A generalist reader can use it to see which selection techniques generalize across datasets and how class imbalance reshapes the rankings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Domain-specific method reimplementations lack external validation; SigAPI's top-three ranking may reflect the authors' reconstruction, not the published method.","rationale":"The reader's weakest assumption is exactly the faithfulness of the six domain-specific reimplementations, and my analysis identifies this as the most load-bearing condition for the central claim. If SigAPI or another domain-specific method is implemented differently from the published algorithm, the empirical ranking (which names SigAPI in the top three) does not describe the published methods; it describes the authors' reconstructions. The paper provides only internal review as evidence of fidelity, which is not an external check. This does not prove the implementations are wrong, but it leaves a concrete, verifiable gap in the central comparison. I also note the ABC/LR inclusion contradiction as supporting evidence that the method inventory is not fully consistent, but the reimplementation-fidelity concern is the primary one because it directly targets the headline result. The reader's conditional verdict remains appropriate: the paper could be accepted if the repository and reimplementation checks are supplied, but the current evidence is insufficient to fully support the empirical claim. Therefore, no change to the reader's verdict is needed.","tokens_in":15029,"tokens_out":3698,"duration_ms":44904,"concrete_test":"Run the SigAPI implementation from the MH-FSF repository on the same Drebin-based permission/API dataset described in the original SigAPI paper (Galib & Hossain 2020), using the original paper's stated feature-selection procedure, and compare the ranking/selected API-call features with those reported in the original publication. If the selected feature set differs materially (e.g., Jaccard similarity < 0.9) or the reproduced classifier metrics do not match the original paper's reported range, the framework's domain-specific comparison is not validated. Repeat for at least one more domain-specific method (e.g., SemiDroid) to confirm the pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim ('LASSO, RFE, and SigAPI emerge as most reliable') depends on the six domain-specific reimplementations being faithful to their published originals. Section III states only that each reproduction 'undergo review and technical evaluation by at least three researchers' (Section III, framework development paragraph); no comparison is given between reproduced outputs and the feature sets or performance numbers reported in the original papers (SemiDroid [4], RFG [7], JOWNDroid [8], MT [9], SigPID [10], SigAPI [6]). SigAPI is specifically named in the headline result, so an implementation drift in SigAPI would directly change the paper's main conclusion. The paper's own internal inconsistency—Section III says ABC and LR 'were excluded from the current analysis,' yet Table V and Figures 2–4 include ABC and LR—shows the reported method set is not fully reliable as a record of what was actually evaluated. Without an external anchor (e.g., reproducing original papers' selected feature subsets or metrics), the comparison between domain-specific and classical methods measures the authors' reconstructions, not the published methods.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MH-FSF, a modular framework for feature selection evaluation, and uses it to compare 17 feature selection methods (11 classical and 6 domain-specific) on 10 publicly available Android malware datasets, using three classifiers and metrics including MCC. The authors report that LASSO, RFE, and SigAPI are the most reliable methods across balanced and imbalanced datasets, and they argue that class balancing is critical for obtaining consistent feature-selection performance. The framework and full results are said to be publicly available on GitHub.","tokens_in":15134,"tokens_out":4518,"duration_ms":50909,"significance":"If the claims hold, the paper would provide a valuable reproducible benchmark for feature selection in Android malware detection, with a public, extensible framework integrating both classical and domain-specific methods. The authors are explicit about using public datasets and default scikit-learn configurations, and they provide MCC heatmaps and per-method boxplots, which are welcome additions. However, the empirical conclusions currently rest on several load-bearing methodological details that are either unspecified or internally inconsistent, so the benchmark's value will be fully realized only after these points are resolved.","major_comments":[{"comment":"The text states that ABC and LR were 'excluded from the current analysis' because of inferior performance, yet Table V and Figures 2-4 report results for both ABC and LR. This is a direct internal contradiction about which 17 methods were actually evaluated; the authors must reconcile the roster and remove or correct the exclusion statement.","section":"Section III, Table V, Figures 2-4"},{"comment":"The experimental protocol does not state whether feature selection is performed inside or outside the cross-validation loop. The pipeline in Figure 1 places 'Features Selection' before 'Model Training and Evaluation,' and Section V says the three classifiers are used to evaluate 'the datasets resulting from feature selection,' suggesting the feature subsets were derived from the full datasets and then evaluated with stratified 5-fold CV. If that is the case, the test folds have influenced feature selection, producing optimistically biased metrics and invalidating the relative ranking of methods. The paper must specify the exact placement of feature selection relative to CV; if it occurs before CV, the experiments should be rerun with feature selection inside each training fold.","section":"Section V and Figure 1"},{"comment":"The paper's central conclusion about the benefits of class balancing cannot be verified because no balancing method is described. Section V only mentions stratified cross-validation and the use of default scikit-learn configurations; the procedure used to produce the 'Balanced Per Class' counts in Table IV (e.g., random undersampling of the majority class, SMOTE, or another resampling scheme) is never stated. This omission is a reproducibility blocker for all balanced-dataset results, which are a key component of the empirical claims.","section":"Section V and Table IV"},{"comment":"The fidelity of the six domain-specific reimplementations is supported only by an internal three-researcher review, with no quantitative comparison against the feature sets or performance numbers reported in the original papers (SemiDroid [4], RFG [7], JOWNDroid [8], MT [9], SigPID [10], SigAPI [6]). Because SigAPI is named in the headline result (Section VI), the ranking of domain-specific versus classical methods measures the authors' reconstructions unless external validation is provided. Include a comparison of reproduced outputs (e.g., selected features or classification performance on the original datasets) or explicitly limit the claims to the implementations as provided.","section":"Section III and Section VI"}],"minor_comments":[{"comment":"The caption contains a typo: 'distribuition' should be 'distribution'.","section":"Figure 2 caption"},{"comment":"The figure contains 'KroinoDroid R' while the text and tables use 'KronoDroid R'; please unify the spelling.","section":"Figure 1"},{"comment":"The sentence mentioning FSDroid as an additional framework method is confusing because FSDroid is not listed in Table II and is never defined; please explain or remove the reference.","section":"Section III"},{"comment":"The method name 'ANOV A' appears with an extra space throughout (e.g., Table II and Table V); this should be corrected to 'ANOVA'.","section":"Tables and text"},{"comment":"The exact versions of the public datasets are not given; for example, KronoDroid has multiple releases, and the paper should identify the specific versions used, even if they are available in the repository.","section":"Section V and Table IV"},{"comment":"The phrase 'in presence of class imbalance' should be 'in the presence of class imbalance'; similar small grammatical fixes are needed in several places.","section":"Section VII"}],"recommendation":"major_revision","confidential_remarks":"The framework's public GitHub repository and the authors' explicit effort to use public datasets and default library configurations are positive features of the submission. However, the internal contradiction about which methods were evaluated, the unspecified balancing procedure, and the unclear placement of feature selection relative to cross-validation are load-bearing and should be addressed before publication. Please ask the authors to provide detailed responses to these points, including precise experimental protocol text and, if applicable, rerun results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: MH-FSF is real subfield infrastructure, not a new method. The genuinely new contribution is the harness plus the 17-method × 10-dataset × 3-classifier result matrix on public Android malware datasets, with balanced and complete variants. That is broader than anything in the cited literature, and if the GitHub repo is as complete as claimed, it gives the community a common testbed.\n\nWhat the paper does well: the protocol is mostly stated — scikit-learn 1.5.2 defaults, stratified 5-fold CV, MCC reported alongside F1 and recall. The headline ranking (LASSO and RFE robust; PCA, ReliefF, and SigPID brittle under imbalance) is plausible and consistent with prior work. The paper is honest about fragmentation in the field, and the three-researcher internal review for reimplementations is better than nothing.\n\nSoft spots, in order of severity:\n\n1. Internal contradiction. Section III says ABC and LR were excluded from the current analysis, yet Table V and Figures 2–4 report results for both. That is a factual inconsistency in the method roster and must be fixed.\n2. The balancing procedure is never described. \"Class balancing\" appears in the pipeline diagram, but not how it was done — downsampling, upsampling, SMOTE? This changes every balanced result.\n3. No seeds or significance tests. Many scores are within 0.01 of each other, so the ranking may partly reflect noise.\n4. The six domain-specific reimplementations lack external validation. The comparison measures the authors' reconstructions unless they show reproduced feature subsets or match the original papers' reported metrics. Since SigAPI is in the top-three headline, this is load-bearing.\n\nThe stress-test note's circularity concern does not land: there is no derivation, results are on held-out folds, and the datasets are public. The reimplementation issue is the real weakness, and the ABC/LR contradiction shows the paper's record of what was actually run is not fully reliable.\n\nWho this is for: anyone benchmarking feature selection in Android malware, or building evaluation harnesses. It deserves a serious referee — the artifact fills a gap — but the revision should resolve the contradiction, document balancing and seeds, and anchor at least one or two reimplementations against their original published outputs.\n\nRecommendation: send to peer review; require those fixes.","headline":"A useful benchmarking artifact whose result matrix is broader than anything in the cited literature, but the central ranking depends on unvalidated reimplementations and the paper contains an internal contradiction about which methods were actually evaluated.","tokens_in":15780,"tokens_out":2760,"would_cite":true,"duration_ms":33524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a unified, publicly checkable framework for feature selection evaluation can replace fragmented, proprietary benchmarking in Android malware detection, and that under identical conditions LASSO, RFE, and SigAPI are…","keywords":["feature selection","Android malware detection","benchmarking","reproducibility","class imbalance","LASSO","domain-specific methods","evaluation framework"],"falsifier":"Re-run the six domain-specific methods on the datasets named in their original papers and compare the selected feature sets or reported F1 and recall values with those in the original publications; if the reproduced outputs differ materially, the MH-FSF comparison measures the reconstructions rather than the published methods.","tokens_in":14728,"feed_emoji":"📊","tokens_out":4351,"duration_ms":46396,"temperature":0.7,"pith_summary":"This paper argues that feature selection research in Android malware detection has been held back by two fixable problems: evaluations rely on private datasets, and new methods are compared against only a handful of similar baselines. To fix this, the authors introduce MH-FSF, a modular, publicly available framework that reimplements 17 feature selection methods—11 classical and 6 domain-specific—and runs them through the same data preparation, model training, and evaluation pipeline on 10 public Android malware datasets. The comparative results show that no method wins everywhere: performance shifts between balanced and imbalanced data, and the most consistently reliable methods are LASSO, RFE, and SigAPI. If the framework is taken up, the field gains a common yardstick for new methods and a checkable record of how published techniques behave outside their original private datasets.","feed_headline":"One open platform ranks 17 feature selectors on 10 malware datasets","feed_subtitle":"LASSO, RFE, and SigAPI hold up best; the code and data are public for rechecking.","key_machinery":"The load-bearing mechanism is the MH-FSF pipeline, a four-stage architecture: data manipulation (NaN and duplicate removal, balancing, sampling), feature selection, classifier training and evaluation, and visualization. Each feature selection method is isolated as an independent module with a standard interface, so new techniques can be added without structural changes. The comparison is carried by this uniformity: all 17 methods feed the same reduced datasets to the same three classifiers (KNN, Random Forest, SVM) under stratified 5-fold cross-validation, and performance is aggregated over accuracy, precision, recall, F1, ROC-AUC, and MCC.","core_discovery":"On the paper's own terms, the central discovery is that a single reproducible evaluation platform can change what we believe about feature selection in this domain. Under identical conditions across 10 datasets and three classifiers, classical methods LASSO and RFE, together with the API-focused domain-specific method SigAPI, deliver the highest average F1 and recall with the least variability, while methods like PCA, ReliefF, and SigPID degrade markedly on imbalanced data. The paper further finds that several domain-specific methods, originally validated on single proprietary datasets, do not generalize: JOWMDroid and SigPID perform poorly outside their original feature types, suggesting that earlier isolated comparisons overstated their value. The framework claim is that MH-FSF itself—with 17 implementations, 10 public datasets, and all results released—provides the infrastructure to make such comparisons routine and reproducible.","pith_inferences":["Going beyond the paper: if the framework is applied to other domains it claims to support—network traffic, biomedical data, fraud detection—the same protocol could expose which classical methods are domain-robust, something the current Android-only evaluation cannot show.","The paper does not compare its reimplementations against the original methods' reported feature sets, so a fair reading is that its rankings measure the reconstructions unless such validation is added.","A testable extension is to treat dataset balancing as an intervention: the MCC heatmaps suggest balancing helps most methods, so a controlled study varying only the balancing rule could identify which selection methods benefit most."],"forward_implications":["Researchers can now check any of the 17 methods against the same 10 public datasets, so new proposals need not rely on private data to claim an improvement.","The ranking implies that LASSO and RFE are the safe default choices for Android malware detection, with SigAPI the strongest domain-specific option on API-call datasets.","Because balancing changes which methods win, the paper's results imply that class imbalance should be reported and treated as a first-class experimental variable in future feature selection studies.","Methods validated only on one private dataset, such as JOWMDroid and SigPID, should be re-evaluated on diverse public data before being adopted."],"supporting_citations":[{"why":"Defines SemiDroid, one of the six domain-specific methods reproduced in the framework.","marker":"[4]"},{"why":"Defines SigAPI, the API-focused method the paper identifies as most reliable among domain-specific techniques.","marker":"[6]"},{"why":"Defines RFG, a domain-specific method included in the comparison.","marker":"[7]"},{"why":"Defines JOWMDroid, a domain-specific method whose poor generalization the paper attributes to single-dataset validation.","marker":"[8]"},{"why":"Defines the Multi-Tiered feature selection model included as a domain-specific method.","marker":"[9]"},{"why":"Defines SigPID, the permission-focused method that underperforms on API-dominated datasets.","marker":"[10]"},{"why":"Supplies the classifiers and default configurations used to evaluate every reduced dataset.","marker":"[45]"},{"why":"Points to the public MH-FSF repository that makes implementations, datasets, and full results available for reproducibility.","marker":"[34]"}],"fun_headline_variants":["Open framework ranks feature selectors on 10 datasets","LASSO, RFE, SigAPI win in reproducible benchmark","MH-FSF: 17 selectors, 10 datasets, one verdict","Reproducible feature selection benchmark for malware","Which feature selector works best? New framework says"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's ranking stands on the assumption that its implementations of the six domain-specific methods faithfully reproduce the published methods, since it reports no comparison of the reproduced feature sets or scores against the originals.","fun_headline_variants_meta":{"raw":{"variants":["Open framework ranks feature selectors on 10 datasets","LASSO, RFE, SigAPI win in reproducible benchmark","MH-FSF: 17 selectors, 10 datasets, one verdict","Reproducible feature selection benchmark for malware","Which feature selector works best? New framework says"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1162,"prompt_tokens":896,"completion_tokens":266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":185}},"tokens_in":512,"tokens_out":266,"duration_ms":3823,"temperature":1.0,"reasoning_tokens":185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:09:35.390251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six domain-specific methods on the datasets named in their original papers and compare the selected feature sets or reported F1 and recall values with those in the original publications; if the reproduced outputs differ materially, the MH-FSF comparison measures the reconstructions rather than the published methods.","supporting_citations":[{"cited_title":"SemiDroid: A Behavioral Malware Detector Based on Unsupervised Machine Learning Techniques Using Feature Selection Approaches,","cited_arxiv_id":null,"evidence_quote":"Defines SemiDroid, one of the six domain-specific methods reproduced in the framework."},{"cited_title":"Significant API Calls in Android Malware Detection (Using Feature Selection Techniques and Correlation Based Feature Elimination),","cited_arxiv_id":null,"evidence_quote":"Defines SigAPI, the API-focused method the paper identifies as most reliable among domain-specific techniques."},{"cited_title":"Automated Malware Detection in Mobile App Stores Based on Robust Feature Generation,","cited_arxiv_id":null,"evidence_quote":"Defines RFG, a domain-specific method included in the comparison."},{"cited_title":"JOWMDroid: Android Malware Detection Based on Feature Weighting with Joint Optimization of Weight-Papping and Classifier Parameters,","cited_arxiv_id":null,"evidence_quote":"Defines JOWMDroid, a domain-specific method whose poor generalization the paper attributes to single-dataset validation."},{"cited_title":"A Multi-Tiered Feature Selection Model for Android Malware Detection Based on Feature Discrimination and Information Gain,","cited_arxiv_id":null,"evidence_quote":"Defines the Multi-Tiered feature selection model included as a domain-specific method."},{"cited_title":"SigPID: Significant Per- mission Identification for Android Malware Detection,","cited_arxiv_id":null,"evidence_quote":"Defines SigPID, the permission-focused method that underperforms on API-dominated datasets."},{"cited_title":"Scikit-learn: Machine Learning in Python,","cited_arxiv_id":null,"evidence_quote":"Supplies the classifiers and default configurations used to evaluate every reduced dataset."},{"cited_title":"MH-FSF: um Framework para Reproduc ¸˜ao, Experimentac ¸˜ao e Avaliac ¸˜ao de M ´etodos de Selec ¸˜ao de Caracter ´ısticas,","cited_arxiv_id":null,"evidence_quote":"Points to the public MH-FSF repository that makes implementations, datasets, and full results available for reproducibility."}],"review_version":1}