Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

This paper argues that large language models, prompted with feature names and summary statistics, can select malware-detection features as effectively as classical statistical methods while adding interpretability and stability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:42 UTC pith:VLNGJFEY

load-bearing objection A useful domain extension of LLM-Select, but the 'zero-shot' claim does not survive contact with the prompts: class-conditional statistics do much of the work. the 3 major comments →

arxiv 2602.09634 v1 pith:VLNGJFEY submitted 2026-02-10 cs.LG cs.CR

LLM-FS: Zero-Shot Feature Selection for Effective and Interpretable Malware Detection

Naveen Gill , Ajvad Haneef K , Madhu Kumar S D This is my paper
classification cs.LG cs.CR
keywords feature selectionlarge language modelszero-shot learningmalware detectioninterpretabilitytabular datahigh-dimensional dataprompt-based scoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that large language models can perform feature selection for malware detection without any training on the dataset, by reading each feature's name and summary statistics and returning an importance score. If true, this matters because classical feature-selection methods either ignore semantics, depend on a trained model, or require repeated retraining; an LLM could instead provide an interpretable, stable ranking with natural-language reasons. On a 934k-sample, 2,381-feature malware benchmark, the authors report that the LLM-chosen top-341 features match or nearly match traditional selectors across Random Forest, Extra Trees, MLP, and KNN. The accompanying caveat is that the prompt includes class-conditional statistics derived from labels, so the method is not purely label-free; the paper flags that these must be computed on the training split.

Core claim

The paper's central claim is that a zero-shot LLM, given a per-feature prompt containing the feature's name, global statistics, and class-conditional statistics, plus the task 'classify whether a file is malware or benign,' can assign importance scores whose top 341 features support classifiers that are competitive with classifiers trained on features chosen by Extra Trees, Chi-Squared, ANOVA, Mutual Information, Variance Threshold, and other classical selectors. The authors take this as evidence that LLMs inject semantic, domain-grounded knowledge into tabular feature selection, not just statistical ranking, and that this yields interpretable, stable, classifier-agnostic feature subsets.

What carries the argument

The load-bearing machinery is a per-feature structured descriptor D(f_j) = {mean, standard deviation, median, min, max, IQR, class-conditional means and standard deviations, mean difference}, formatted into a prompt with task context and sent to an LLM with deterministic decoding; the returned score s_j in [0,1] ranks the feature, with invalid outputs defaulting to a neutral 0.5. This combines a hand-computed statistical summary with the LLM's pretrained semantic associations, and it is what lets the method claim zero-shot knowledge-driven selection.

Load-bearing premise

The method's knowledge-driven advantage depends on the LLM scoring features from semantic understanding rather than from the label-derived averages and spreads included in each prompt; if those numbers alone explain the rankings, the zero-shot claim loses its force.

What would settle it

Regress the LLM's importance scores on the per-feature absolute t-statistic of the class-mean difference; if the rank correlation is, say, above 0.9, the LLM is transcribing label-derived statistics rather than adding semantic knowledge.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, feature selection can be performed once and reused across classifiers, since the LLM's ranking is largely classifier-agnostic; operational pipelines would save repeated re-selection costs.
  • Analysts could get natural-language rationales linking a feature's statistical separability to behavioral hypotheses, such as high section entropy suggesting packing or obfuscation, which can support audit and trust.
  • A hybrid pipeline that runs a cheap statistical filter first and then has the LLM re-rank a shortlist would retain most of the interpretability benefit while cutting API cost.
  • The approach's ceiling depends on the semantic quality of feature names; obfuscated or purely numeric features would erode the zero-shot advantage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One untested possibility is that the LLM is functioning as a re-ranker of the label-derived statistics already in the prompt: if the scores are nearly monotone in, say, the class-mean difference, the semantic-knowledge story is overstated and the contribution reduces to a statistical prefilter with a language-model front end.
  • A genuinely label-free variant would drop class-conditional statistics and rely only on feature names, global statistics, and task context; comparing that variant's accuracy against Random Selection would directly measure the value of LLM semantics.
  • The interpretability claim could be tested by collecting the LLM's rationales and checking whether they correctly identify which side of a feature's distribution corresponds to malware; errors there would reveal whether rationales are post-hoc text or genuine explanation.
  • Because the paper uses a fixed top-k (341) chosen by preliminary experiments, the comparison conflates the quality of the LLM ranking with the quality of the k choice; measuring performance across multiple k values would separate the two.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LLM-FS, a feature-selection framework for high-dimensional malware detection. For each feature, the method computes global and class-conditional statistics from the labeled training data, packages them into a prompt together with the feature name and task description, and asks an LLM to output an importance score in [0,1]. The top-341 features are then used to train Random Forest, Extra Trees, MLP, and KNN classifiers. The authors compare several LLMs (GPT-4.0, GPT-4.0-mini, GPT-5.0-mini, Gemini-2.5 variants) against traditional filter/wrapper/embedded feature-selection baselines on the EMBOD dataset (a fusion of EMBER and BODMAS), reporting accuracy, precision, recall, F1, AUC, MCC, and runtime. They conclude that LLM-guided feature selection is competitive with traditional methods while providing interpretability, stability, and reduced dependence on labeled data.

Significance. If the central claim were fully supported, the paper would make a useful contribution: a knowledge-guided feature-selection method that works when labels are scarce could be valuable in security domains, and the systematic comparison across many classifiers and metrics is a strength. The paper also honestly acknowledges several threats to validity, including the training-split requirement for class-conditional statistics and distributional shift from dataset fusion. However, the most distinctive claims — 'zero-shot' selection and 'reduced dependence on labeled data' — are undermined by the method itself, which feeds class-conditional statistics computed from labeled data into the prompt. Because no ablation isolates the contribution of LLM semantic knowledge from the statistical signal already present in the prompt, the paper does not currently demonstrate that LLM reasoning, rather than a statistical prefilter, drives performance. The lack of variance estimates and the unexamined choice of k=341 also weaken the empirical case. The contribution is potentially salvageable as a hybrid statistical-plus-semantic feature selector, but the present claims need substantial revision and add

major comments (3)
  1. [Abstract and Section III-A2, Algorithm 1, Eqs. (6)-(11)] The abstract claims LLM-FS works 'using only feature names and task descriptions,' but Algorithm 1 and Eq. (11) show that the prompt actually contains class-conditional means, standard deviations, and Δμ_j computed from labeled training data (Eqs. (6)-(10)). This is not the zero-shot setting claimed, and it undermines 'reduced dependence on labeled data.' The paper needs an ablation that isolates three conditions: (a) feature names + task description only, (b) global statistics added, (c) class-conditional statistics added. It also needs a non-LLM control that uses the same statistics directly, e.g., ranking by |Δμ_j|/σ_j or ANOVA F-score, to show the LLM adds value beyond the statistics already in the prompt. Section IV-E mentions the training-split requirement but does not provide this ablation.
  2. [Section IV-D, Table II] The number of selected features k=341 is described as chosen 'based on preliminary experiments that indicated this number provided a good balance between dimensionality reduction and model performance,' but no details of those experiments are given. If k was tuned on the test set or selected after observing the comparison results, the reported numbers are optimistic. Please report how k was chosen, which data split was used for the choice, and include a sensitivity analysis over k (e.g., k=100, 200, 341, 500) to show the conclusions are not artifacts of this specific choice.
  3. [Sections IV-A and IV-D, Table II] The paper claims 'stability across runs' as an advantage, but Table II reports a single run for each FS/classifier combination, and no variance, confidence intervals, or significance tests are provided over repeated train/test splits or random seeds. The qualitative discussion in Section IV-E ('LLM-FS showed competitive or lower variance across runs') is not backed by experimental evidence in the paper. Please report repeated trials (e.g., 5-10 different seeds or data splits) with mean±std for at least the main comparisons, and perform a statistical test (e.g., Wilcoxon) for the claim that LLM-FS matches traditional methods.
minor comments (6)
  1. [Abstract and Table II] Model naming is inconsistent: the abstract mentions 'GPT-5.0' while the table reports 'gpt-5-mini'; please clarify the exact model versions and API identifiers used.
  2. [Section III-A1 and Table II] Table II includes 'Sequential Attention' and 'Correlation-threshold' but the methodology subsection does not define 'Sequential Attention' (only Sequential Feature Selection) or describe how the correlation threshold was set. Please align the list of baselines between the text and the table.
  3. [Section III-A, Eq. (3)] Equation (3) writes the feature-selection objective as an arg max over X', but X' is the reduced feature matrix rather than a subset index set; the notation is confusing and should be rewritten, e.g., as arg max over feature subsets S with |S|=k.
  4. [Section IV-A] The classifiers are said to use 'default hyperparameters,' but the specific library versions and defaults are not reported. Since MLP and KNN performance can be sensitive to hyperparameters, please provide the exact configurations (or a link to code) for reproducibility.
  5. [Table II, runtime column] Some runtime values are highly counterintuitive, e.g., KNN with VarianceThreshold takes 20,343s while KNN with Tree-Based takes 118s. Clarify whether runtime includes feature-selection time, and explain the large discrepancies, which may reflect API latency or implementation artifacts.
  6. [Throughout] There are many typos and infelicities ('ANOV A', 'malcontent', 'reliability on statistical filters', 'dived into', 'postions'). The paper would benefit from careful proofreading.

Circularity Check

1 steps flagged

Zero-shot claim is confounded by label-derived statistics in the prompt; core benchmark comparison remains independent.

specific steps
  1. fitted input called prediction [Abstract; Section III-A2, Eqs. (10)-(12), Algorithm 1; Section IV-E]
    "using only feature names and task descriptions (Abstract); Algorithm 1 input: X∈R^{n×d}, y∈{0,1}^n; Eq. (11): D(f_j)= {μ_j, σ_j, median, min, max, IQR, μ_j^(1), μ_j^(0), σ_j^(1), σ_j^(0), Δμ_j, samples}"

    The paper's central zero-shot claim is that LLMs select features using only feature names and task descriptions. But the LLM score s_j = Φ(D(f_j), C) is computed from a descriptor that contains class-conditional means, class-conditional standard deviations, and Δμ_j = μ_j^(1) − μ_j^(0), all computed from the labels y in Eqs. (6)-(10). Thus the feature-importance 'prediction' is, by construction, a function of label-derived statistics; any performance attributed to zero-shot semantic knowledge is confounded with a statistical prefilter. The paper itself acknowledges in Section IV-E that these statistics must be computed on the training split to avoid label leakage, confirming they are label-dependent. No ablation separates the LLM's semantic contribution from the statistics, so the claimed

full rationale

The main empirical comparison is self-contained and not circular: LLM-FS is benchmarked against traditional FS methods (Extra Trees, Chi-squared, ANOVA, etc.) with reported metrics, and the paper does not rely on self-citations to support its core performance claims. However, the paper's headline 'zero-shot' claim is undermined by its own methodology: the prompt includes class-conditional statistics computed from the training labels, so the LLM is not selecting features from feature names alone. This is closer to an overstatement/confound than a full mathematical equivalence, because the LLM's score is not shown to equal the input statistics and feature-name semantics could contribute. The paper also flags the training-split requirement but does not ablate the statistics. Given the central 'reduced dependence on labeled data' claim is partially circular by construction while the benchmark comparison retains independent content, a score of 4 is appropriate.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claim rests on the EMBOD dataset, a fixed k=341 determined by undisclosed preliminary experiments, and the assumption that LLM scores reflect semantic knowledge rather than the statistical descriptors in the prompt. No new entities are invented.

free parameters (2)
  • number of selected features k = 341
    Chosen based on 'preliminary experiments that indicated this number provided a good balance' (Section IV-D). No details of these experiments are provided, so the choice could be tuned to the reported results.
  • test-train split ratio = 80:20
    A conventional split, but no cross-validation is used, so results depend on this single split.
axioms (3)
  • domain assumption The EMBOD dataset is a valid fusion of EMBER and BODMAS, with features extracted using LIEF 0.9.0.
    The paper relies entirely on this dataset for its empirical conclusions, but provides no details about how the fusion was performed, how class labels are balanced, or how overlap between the original datasets was handled.
  • domain assumption The LLM APIs respond deterministically at temperature=0.
    The paper claims stability due to temperature=0, but also admits 'some variability persists due to model updates and API nondeterminism.'
  • domain assumption Feature names from the EMBOD dataset carry semantic meaning that LLMs can exploit.
    This is the core assumption behind the method; the paper acknowledges that obfuscated or synthetic feature names would reduce the advantage.

pith-pipeline@v1.3.0-alltime-deepseek · 12019 in / 6524 out tokens · 49058 ms · 2026-08-03T02:42:01.909029+00:00 · methodology

0 comments
read the original abstract

Feature selection (FS) remains essential for building accurate and interpretable detection models, particularly in high-dimensional malware datasets. Conventional FS methods such as Extra Trees, Variance Threshold, Tree-based models, Chi-Squared tests, ANOVA, Random Selection, and Sequential Attention rely primarily on statistical heuristics or model-driven importance scores, often overlooking the semantic context of features. Motivated by recent progress in LLM-driven FS, we investigate whether large language models (LLMs) can guide feature selection in a zero-shot setting, using only feature names and task descriptions, as a viable alternative to traditional approaches. We evaluate multiple LLMs (GPT-5.0, GPT-4.0, Gemini-2.5 etc.) on the EMBOD dataset (a fusion of EMBER and BODMAS benchmark datasets), comparing them against established FS methods across several classifiers, including Random Forest, Extra Trees, MLP, and KNN. Performance is assessed using accuracy, precision, recall, F1, AUC, MCC, and runtime. Our results demonstrate that LLM-guided zero-shot feature selection achieves competitive performance with traditional FS methods while offering additional advantages in interpretability, stability, and reduced dependence on labeled data. These findings position zero-shot LLM-based FS as a promising alternative strategy for effective and interpretable malware detection, paving the way for knowledge-guided feature selection in security-critical applications

Figures

Figures reproduced from arXiv: 2602.09634 by Ajvad Haneef K, Madhu Kumar S D, Naveen Gill.

Figure 1
Figure 1. Figure 1: Architecture of proposed LLM-FS framework [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of samples in the EMBOD dataset [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Heatmap comparing LLM-based and traditional FS Methods across multiple classifiers: (a) Random Forest, (b) Extra Trees (c) KNN , and (d) MLP. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classification

    cs.LG 2026-07 conditional novelty 6.0

    SMETA-ZSL combines contrastive LLM prototypes, episodic meta-alignment, and Z-score gating to beat prior generalized zero-shot methods by ~10.8 harmonic-mean points under strict open-set inductive conditions on seven ...

Reference graph

Works this paper leans on

26 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    A review on large language models: Architectures, applications, taxonomies, open issues and challenges,

    M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, and S. Azam, “A review on large language models: Architectures, applications, taxonomies, open issues and challenges,”IEEE access, vol. 12, pp. 26839–26874, 2024

    2024

  2. [2]

    Enhancing malware detection with fea- ture selection and scaling techniques using machine learning models,

    R. Hasan, B. Biswas, M. Samiun, M. A. Saleh, M. Prabha, J. Akter, F. H. Joya, and M. Abdullah, “Enhancing malware detection with fea- ture selection and scaling techniques using machine learning models,” Scientific Reports, vol. 15, no. 1, p. 9122, 2025

    2025

  3. [3]

    The impact of feature selection on malware classification using chi-square and machine learning,

    A. F. Rasheed, M. Zarkoosh, and S. S. Al-Azzawi, “The impact of feature selection on malware classification using chi-square and machine learning,” in2023 9th International Conference on Computer and Communication Engineering (ICCCE), pp. 211–216, IEEE, 2023

    2023

  4. [4]

    Feature selection for malware detection based on reinforcement learning,

    Z. Fang, J. Wang, J. Geng, and X. Kan, “Feature selection for malware detection based on reinforcement learning,”IEEE Access, vol. 7, pp. 176177–176187, 2019

    2019

  5. [5]

    Analysis and comparison of feature selection methods towards performance and stability,

    M. C. Barbieri, B. I. Grisci, and M. Dorn, “Analysis and comparison of feature selection methods towards performance and stability,”Expert Systems with Applications, vol. 249, p. 123667, 2024

    2024

  6. [6]

    A review of feature selection methods for actual evapotranspiration prediction,

    C. M. Liyew, S. Ferraris, E. Di Nardo, and R. Meo, “A review of feature selection methods for actual evapotranspiration prediction,”Artificial Intelligence Review, vol. 58, no. 10, p. 292, 2025

    2025

  7. [7]

    Stability of feature selection algorithm: A review,

    U. M. Khaire and R. Dhanalakshmi, “Stability of feature selection algorithm: A review,”Journal of King Saud University-Computer and Information Sciences, vol. 34, no. 4, pp. 1060–1073, 2022

    2022

  8. [8]

    Llm-select: Feature se- lection with large language models,

    D. P. Jeong, Z. C. Lipton, and P. Ravikumar, “Llm-select: Feature se- lection with large language models,”arXiv preprint arXiv:2407.02694, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Knowledge-driven feature selection and engineering for genotype data with large language models,

    J. Lee, S. Yang, J. Y . Baik, X. Liu, Z. Tan, D. Li, Z. Wen, B. Hou, D. Duong-Tran, T. Chen,et al., “Knowledge-driven feature selection and engineering for genotype data with large language models,”AMIA Summits on Translational Science Proceedings, vol. 2025, p. 250, 2025

    2025

  10. [10]

    Llm-fe: Automated fea- ture engineering for tabular data with llms as evolutionary optimizers,

    N. Abhyankar, P. Shojaee, and C. K. Reddy, “Llm-fe: Automated fea- ture engineering for tabular data with llms as evolutionary optimizers,” arXiv preprint arXiv:2503.14434, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Ember: an open dataset for train- ing static pe malware machine learning models,

    H. S. Anderson and P. Roth, “Ember: an open dataset for train- ing static pe malware machine learning models,”arXiv preprint arXiv:1804.04637, 2018

    Pith/arXiv arXiv 2018

  12. [12]

    Bodmas: An open dataset for learning based temporal analysis of pe malware,

    L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, “Bodmas: An open dataset for learning based temporal analysis of pe malware,” in2021 IEEE Security and Privacy Workshops (SPW), pp. 78–84, IEEE, 2021

    2021

  13. [13]

    Stacking llm models’ predictions for feature selection in anomaly classification,

    R. Santana, “Stacking llm models’ predictions for feature selection in anomaly classification,” in2024 International Conference on Sustain- able Technology and Engineering (i-COSTE), pp. 01–06, IEEE, 2024

    2024

  14. [14]

    Initial-llm: A large language model-guided metaheuristic framework for enhanced feature selection in clinical decision support systems,

    Z. Wang, Y . Liang, W. Sun, C. Xu, Y . Zhou, and Y . Zhang, “Initial-llm: A large language model-guided metaheuristic framework for enhanced feature selection in clinical decision support systems,” inInternational Conference on Health Information Science, pp. 27–35, Springer, 2024

    2024

  15. [15]

    A review of feature selection methods for machine learning-based disease risk prediction,

    N. Pudjihartono, T. Fadason, A. W. Kempa-Liehr, and J. M. O’Sullivan, “A review of feature selection methods for machine learning-based disease risk prediction,”Frontiers in bioinformatics, vol. 2, p. 927312, 2022

    2022

  16. [16]

    Malware analysis and detection using machine learning algorithms,

    M. S. Akhtar and T. Feng, “Malware analysis and detection using machine learning algorithms,”Symmetry, vol. 14, no. 11, p. 2304, 2022

    2022

  17. [17]

    An introduction to variable and feature selection,

    I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,”Journal of machine learning research, vol. 3, pp. 1157– 1182, 2003

    2003

  18. [18]

    Chi2: Feature selection and discretization of numeric attributes,

    H. Liu and R. Setiono, “Chi2: Feature selection and discretization of numeric attributes,” inProceedings of the Seventh International Conference on Tools with Artificial Intelligence, pp. 388–391, IEEE, 1995

    1995

  19. [19]

    Optimal ensemble learning based on distinctive feature selection by univariate anova-f statistics for ids,

    S. Shakeela, N. S. Shankar, P. M. Reddy, T. K. Tulasi, and M. M. Sai, “Optimal ensemble learning based on distinctive feature selection by univariate anova-f statistics for ids,”International Journal of Electronics and Telecommunication, 2021. Active feature-selection using ANOV A- F in IDS context

    2021

  20. [20]

    Feature selection based on mu- tual information criteria of max-dependency, max-relevance, and min- redundancy,

    H. Peng, F. Long, and C. Ding, “Feature selection based on mu- tual information criteria of max-dependency, max-relevance, and min- redundancy,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1226–1238, 2005

    2005

  21. [21]

    Correlation-based feature selection for machine learning,

    M. A. Hall, “Correlation-based feature selection for machine learning,” inProceedings of the 17th International Conference on Machine Learning (ICML), pp. 123–134, Morgan Kaufmann, 1999

    1999

  22. [22]

    Random forests,

    L. Breiman, “Random forests,”Machine learning, vol. 45, no. 1, pp. 5– 32, 2001

    2001

  23. [23]

    Extremely randomized trees,

    P. Geurts, D. Ernst, and L. Wehenkel, “Extremely randomized trees,” Machine learning, vol. 63, pp. 3–42, 2006

    2006

  24. [24]

    Wrappers for feature subset selection,

    R. Kohavi and G. H. John, “Wrappers for feature subset selection,” in Artificial Intelligence, vol. 97, pp. 273–324, Elsevier, 1997

    1997

  25. [25]

    A survey on machine learning-based malware detection in executable files,

    J. Singh and J. Singh, “A survey on machine learning-based malware detection in executable files,”Journal of Systems Architecture, vol. 112, p. 101861, 2021

    2021

  26. [26]

    A comprehensive review on malware detection approaches,

    ¨O. A. Aslan and R. Samet, “A comprehensive review on malware detection approaches,”IEEE access, vol. 8, pp. 6249–6271, 2020

    2020