Pith. sign in

REVIEW 4 major objections 5 minor 10 references

Cross-Domain Malware Detection via Probability-Level Fusion of Lightweight Gradient Boosting Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By fusing the predicted probabilities of three lightweight LightGBM models trained on static, behavioral, and memory malware datasets, this paper claims a cross-domain macro F1 of 0.823—better than any single-domain model.

desk verdict The paper's central cross-domain fusion result is not computable as described because the models operate on disjoint feature sets, and the reported F1 is a fit to the very validation set used to pick the fusion weights. read the letter →

arxiv 2509.00476 v1 pith:SR4LICCY submitted 2025-08-30 cs.CR cs.AI

Omar Khalid Ali Mohamed This is my paper · ORCID
classification cs.CRcs.AI
keywords malwaredetectionprobability-levelfusionLightGBMensemblelearningcross-domaingeneralizationstaticfeaturesAPIcallsequencesmemoryanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a malware detector can generalize across very different data sources without training one big model on all features. The proposed recipe is to train three lightweight LightGBM classifiers separately—one on static PE-header features (EMBER), one on API call sequences, one on memory-obfuscation patterns (CIC)—and then combine their predicted probabilities with weights chosen by grid search. The authors report a macro F1 of 0.823 on a validation set pooled from all three domains, where no individual model reaches that level outside its native domain. If the claim holds, cross-domain malware detection can be built from cheap specialists plus a weighted average, which is useful for real-time deployment and for datasets that cannot be concatenated because their feature spaces are incompatible.

What carries the argument

The load-bearing object is Eq. (1), the weighted sum of three model probabilities, with nonnegative weights summing to 1 and optimized by grid search in 0.1 steps over a unified validation set. This fusion step is what turns three domain specialists into one cross-domain detector; it avoids feature concatenation and hard voting, and it is what the ablation studies perturb to show each domain's contribution.

What would settle it

Run the published code on a held-out EMBER sample and pass it to the API-trained and CIC-trained models under the paper's preprocessing. If either model cannot produce a probability because its features are absent, Eq. (1) is undefined for cross-domain samples and the 0.823 score cannot be reproduced as stated. If the code does align features, then recomputing the fused macro F1 on a leave-one-dataset-out split would settle whether the gain is genuine transfer or an artifact of mixing validation sets.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the fused probability y_fused = 0.5·y_ember + 0.4·y_api + 0.1·y_cic, with weights found by exhaustive grid search, achieves macro F1 0.823 on a cross-domain validation set of 28,776 held-out samples. Each single model scores higher on its native set (EMBER 0.965, API 0.867, CIC 1.000), yet the fusion is presented as the detector that generalizes. The result is framed as an extension of ensemble learning: rather than merging features or voting hard labels, the framework preserves each model's confidence and lets the data decide the relative contribution of static, behavioral, and memory evidence.

Load-bearing premise

The entire method assumes every sample can be scored by all three models, even though EMBER, API-call, and CIC-memory datasets use disjoint feature sets; the paper never specifies how a model trained on one feature set produces a probability for a sample described only by the other features.

Editorial extensions

If this is right

  • If the central claim holds, a real-time endpoint could run three LightGBM models on the same sample and combine their scores with fixed weights, keeping inference cheap enough for deployment.
  • Static features carry the most transferable signal (weight 0.5), behavioral features add context (0.4), and memory patterns act as a specialist refinement (0.1).
  • Removing any one of the three datasets reduces macro F1 by 12–18%, so the three domains are not redundant.
  • The weights are not arbitrary: deviations from the optimal combination cost 8–15% macro F1, which justifies the grid-search step rather than simple averaging.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test the paper does not run: train on two domains and test on the third in rotation; that would separate true zero-shot transfer from the advantage that comes from pooling in-domain validation sets.
  • Because the fused score is a convex combination, it cannot beat the best specialist on samples where all three probabilities agree; the practical gain must come from disagreement among the three models, so examining disagreement regions would explain where fusion earns its 0.823.
  • Grid search could be replaced by a logistic regression over the three probabilities; if that matches or exceeds 0.823 with fewer tuned weights, the specific weight values would matter less than the act of calibrating and combining probabilities.
  • The same probability-fusion recipe should transfer to other heterogeneous malware datasets, provided each model's probabilities are well-calibrated; the next natural check is whether weights estimated on one holdout set remain optimal on an independent collection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a lightweight cross-domain malware detection framework that trains separate LightGBM classifiers on three heterogeneous datasets—EMBER (static), API Call Sequences (behavioral), and CIC Obfuscated Memory (memory)—selects top features per dataset, and fuses the predicted probabilities with a weighted sum. Fusion weights are chosen by grid search on a pooled 'cross-domain validation set.' The authors report a fused macro F1 of 0.823, which they interpret as evidence of successful generalization across distinct feature spaces, and they claim full reproducibility. The core problem is that the fusion step is not well-defined: the three datasets have disjoint feature sets, and the manuscript never explains how a model trained on one feature set can produce a probability for a sample from another feature set. In addition, the same validation set is used both to select fusion weights and to report the final metric, making the headline result an in-sample fit. Several dataset split and sample-count inconsistencies further undermine the experimental claims.

Significance. If properly executed, probability-level fusion of lightweight gradient-boosting models is a practical and potentially valuable approach to combining complementary malware-detection signals, and cross-domain robustness is an important goal. The manuscript is clearly written, uses publicly available datasets, and states an intention to provide code and data. However, the central empirical claim—that the fused model generalizes better than any single-domain model—is not supported as presented. The undefined feature-space bridge and the circular validation make the reported 0.823 macro F1 non-reproducible and not a measure of generalization. These are fixable in principle with additional methodological detail and a properly separated evaluation, but they currently undermine the paper's central contribution. The paper also makes claims of 'full reproducibility' without providing a repository link.

major comments (4)
  1. [III-C, Eq. (1); IV-B] Equation (1) requires every sample in the cross-domain validation set to be scored by all three models, but EMBER (2,381 features), API Calls (150 features), and CIC Memory (200 features) have disjoint feature spaces. After top-feature selection (50, 50, 20), the input dimensions are still incompatible. The text does not specify any feature alignment, common embedding, zero-padding, imputation, or other mechanism that would allow the API model to score an EMBER sample or the CIC model to score an API sample. Consequently, the fused probability and the headline macro F1 of 0.823 in Section IV-B are not well-defined quantities under the stated methodology; the experiment cannot be reproduced or executed as described.
  2. [III-C; IV-B] The fusion weights are selected by an exhaustive grid search on the unified cross-domain validation set, and the same validation set is then used to report the fused model's macro F1-score. This is circular: the reported 0.823 is the result of fitting the weights to the evaluation set, not an independent measure of generalization. A proper evaluation requires a separate held-out test set or nested cross-validation for weight selection, and the final metric should be reported on data never used for weight selection. No confidence intervals or error bars are provided either.
  3. [III-A; Table I; IV-B] The validation set sizes in Table I are inconsistent with the stated 80/20 split. EMBER has 639,900 samples, so a 20% split should give 127,980 validation samples, not 63,990. API Calls has 35,100 samples, so a 20% split should give 7,020, not 11,720. CIC Memory is reported with 46,876 validation samples, which is the full dataset size and implies no held-out split. Furthermore, the combined cross-domain validation set is said to contain 28,776 samples, a number not derivable from any of these splits. These inconsistencies raise concerns about data leakage and make it impossible to interpret the individual model performances and the construction of the fusion validation set.
  4. [Table II; IV-B] The comparison in Table II is not apples-to-apples. The per-component macro F1 values (0.965, 0.867, 1.000) are the models' scores on their native validation sets, whereas the fused model's macro F1 (0.823) is computed on the pooled cross-domain validation set. The paper's claim that fusion 'significantly outperforms' individual models is therefore unsupported; it could be true that each individual model performs poorly on the cross-domain set while the fusion improves it, but that is not demonstrated. The table should report each model's macro F1 on the same cross-domain validation set used for the fusion.
minor comments (5)
  1. [References [2], [3]] Reference [2] is titled 'Microsoft Malware Classification Challenge,' but the API Call Sequences dataset is not clearly identified with this source. Reference [3] is described as the CIC Obfuscated Memory dataset, but the cited title is about system-call-based malware detection; please verify and correct the dataset citations.
  2. [Abstract; VI] The abstract and conclusion state that 'all code and data are provided for full reproducibility,' but no repository link, supplementary material, or data availability statement appears in the manuscript. This claim should be substantiated with a URL or appendix.
  3. [IV-B] The composition of the 28,776-sample cross-domain validation set should be specified per dataset, along with label balance. The current description is too vague to assess whether the pooled set is representative.
  4. [Figures 1 and 2] The text references subfigures 1a–1c and 2a–2c, but the figures as printed do not have explicit subfigure labels. Please add labels or revise the references.
  5. [General] There are minor formatting issues, such as an extra space in Reference [4] ('Y . Ke') and inconsistent citation formatting in Reference [8]. Please proofread references.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported cross-domain F1 is measured on the same validation set used to grid-search fusion weights; the headline generalization result is an in-sample fit.

  1. fitted input called prediction [Section III-C (weight grid search) and Section IV-B (Table II, cross-domain F1)]
    "To determine the optimal weights, we constructed a unified fusion validation set from the held-out samples of all three datasets. We performed an exhaustive grid search over w1 and w2 (with w3 = 1 − w1 − w2) in increments of 0.1, selecting the weight combination that maximized the macro F1-score on this diverse validation set. ... The fused model achieved a macro F1-score of 0.823 on the cross-domain validation set, demonstrating successful generalization across distinct feature spaces."

    The same unified validation set is used both as the objective for the grid search (maximize macro F1) and as the set on which the reported fused F1 = 0.823 is measured. The reported number is therefore the in-sample optimum of the search procedure, not an independent out-of-sample prediction. The claim of 'successful generalization' reduces to saying that the selected weights score well on the very set used to select them. No separate test set is held out, so the fused performance is statistically forced to be an optimistic, fitted value rather than evidence of cross-domain generalization.

full rationale

The paper's central claim is that probability-level fusion generalizes better than single-domain models, supported by macro F1 = 0.823 on a 'cross-domain validation set.' However, the text explicitly states that this same validation set was used to perform an exhaustive grid search over fusion weights, selecting the combination that maximizes macro F1. Reporting the F1 on that same set is a textbook fitted-input-as-prediction circularity: the reported metric is the objective used for weight selection, not a held-out evaluation. Because the weights are optimized on the validation set, the improvement over individual models is not demonstrated out of sample; it is, by construction, the best score the grid could achieve on that set. The paper does not rely on self-citations for this step, and the individual model training is independent, but the fusion evaluation itself is circular. Separately, the paper never specifies how each LightGBM model, trained on disjoint feature spaces (EMBER static, API behavioral, CIC memory), can produce a probability for samples from the other datasets, leaving Eq. (1) undefined for the pooled validation set. That is a serious correctness/reproducibility gap, but it is not itself circularity, so it does not further increase the circularity score beyond the in-sample evaluation problem.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several unstated modeling choices: the fusion weights are fitted to the evaluation set, the feature-space alignment is unspecified, and the pooled validation set is treated as a generalization test. These are free parameters and assumptions rather than derived results.

free parameters (3)
  • fusion weights w1, w2, w3 = 0.5, 0.4, 0.1
    Selected by grid search on the cross-domain validation set to maximize macro F1; this is a fitted parameter, not independently derived.
  • top feature counts per dataset = EMBER 50, API 50, CIC 20
    Chosen by hand as a design decision; the paper claims performance is insensitive to +/-20 but no numbers are reported.
  • LightGBM hyperparameters = num_leaves=31, learning_rate=0.05, n_estimators=200
    Fixed by the author to provide a consistent baseline; they are arbitrary choices that affect results.
assumptions (3)
  • domain assumption Each model can produce a malware probability for every sample regardless of feature space.
    Needed for Eq. (1) in Section III-C. The datasets have disjoint features and no alignment is described.
  • domain assumption The pooled validation set, containing held-out samples from all three datasets, is a valid cross-domain test of generalization.
    Used in Section IV-B; this assumption is questionable because individual models are only evaluated on domains they saw in training, while the fused model is evaluated on all domains.
  • domain assumption Weighted sum of uncalibrated probabilities from different models is meaningful without calibration.
    Section III-C assumes probabilities are comparable across models, though no calibration method is applied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Domain Malware Detection via Probability-Level Fusion of Lightweight Gradient Boosting Models." pith.science (2026). https://pith.science/paper/SR4LICCY

@misc{pith2026250900476,
  author       = {Pith},
  title        = {Pith review of: Cross-Domain Malware Detection via Probability-Level Fusion of Lightweight Gradient Boosting Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SR4LICCY}},
  note         = {Machine review of arXiv:2509.00476}
}
read the original abstract

The escalating sophistication of malware necessitates robust detection mechanisms that generalize across diverse data sources. Traditional single-dataset models struggle with cross-domain generalization and often incur high computational costs. This paper presents a novel, lightweight framework for malware detection that employs probability-level fusion across three distinct datasets: EMBER (static features), API Call Sequences (behavioral features), and CIC Obfuscated Memory (memory patterns). Our method trains individual LightGBM classifiers on each dataset, selects top predictive features to ensure efficiency, and fuses their prediction probabilities using optimized weights determined via grid search. Extensive experiments demonstrate that our fusion approach achieves a macro F1-score of 0.823 on a cross-domain validation set, significantly outperforming individual models and providing superior generalization. The framework maintains low computational overhead, making it suitable for real-time deployment, and all code and data are provided for full reproducibility.

Figures

Figures reproduced from arXiv: 2509.00476 by the authors.

Figure 1
Figure 1. Probability distributions for EMBER, API, and CIC [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Confusion matrices for EMBER, API, and CIC models [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

  1. [1]

    EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,

    M. Anderson and P. Roth, “EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models,” in ArXiv preprint arXiv:1804.04637, 2018

    arXiv 2018

  2. [2]

    Mi- crosoft Malware Classification Challenge,

    R. Ronen, M. Radu, C. Feuerstein, E. Yom-Tov, and M. Mancuso, “Mi- crosoft Malware Classification Challenge,” 2018. [Online]. Available: https://arxiv.org/abs/1802.10135

    arXiv 2018

  3. [3]

    A Quantitative Study of Accuracy in System Call-Based Malware Detection,

    D. Canali, A. Lanzi, D. Balzarotti, C. Kruegel, M. Christodorescu, and E. Kirda, “A Quantitative Study of Accuracy in System Call-Based Malware Detection,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA) , 2012, pp. 122-132

    work page 2012

  4. [4]

    LightGBM: A Highly Efficient Gradient Boosting Decision Tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y . Liu, “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 3146-3154

    work page 2017

  5. [5]

    On Combining Classi- fiers,

    J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, “On Combining Classi- fiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, 1998

    work page 1998

  6. [6]

    Multi-source malware detection using feature and decision-level fusion,

    L. Wang, Z. Li, and J. Zhang, “Multi-source malware detection using feature and decision-level fusion,” Computers & Security , vol. 92, p. 101740, 2020

    work page 2020

  7. [7]

    N-grams-based File Signatures for Malware Detection,

    I. Santos, Y . K. Penya, J. Devesa, and P. G. Bringas, “N-grams-based File Signatures for Malware Detection,” in Proceedings of the International Conference on Enterprise Information Systems (ICEIS) , 2009, pp. 317- 320

    work page 2009

  8. [8]

    Effectiveness of API Calls for Malware Detection,

    K. U ˘gurlu, “Effectiveness of API Calls for Malware Detection,” in Proceedings of the International Conference on Cyber Security and Protection of Digital Services (Cyber Security) , 2019, pp. 1-8

    work page 2019

Show all 10 references
  1. [9]

    nox...!? Boosting Malware Detection with Feature Ablation,

    R. Harang and E. M. Rudd, “nox...!? Boosting Malware Detection with Feature Ablation,” in Proceedings of the ACM Workshop on Artificial Intelligence and Security (AISec) , 2020, pp. 43-54

    2020

  2. [10]

    DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket,

    D. Arp, M. Spreitzenbarth, M. H ¨ubner, H. Gascon, and K. Rieck, “DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket,” in Proceedings of the Network and Distributed System Security Symposium (NDSS) , 2014

    2014

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.