{"id":"e1647bc8-7e59-4d46-b4bd-873844b14239","arxiv_id":"2509.00476","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A grid-search weighted average of LightGBM models on EMBER, API-call, and CIC-memory data is reported to reach 0.823 macro F1, but the validation set is both the tuning set and the test set, and the feature alignment is unspecified.","lead":"This paper combines three lightweight malware detectors, trained on static, behavioral, and memory data, by averaging their scores with weights chosen by grid search. The claimed cross-domain gain is not supported because the same data used to pick the weights is also used to report the final score, and the paper never explains how models with different feature spaces can each score the same sample.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fusion step is undefined: no mechanism is specified for applying each LightGBM model to samples with disjoint feature sets, so the fused cross-domain score cannot be computed or reproduced.","rationale":"The reader's weakest assumption identifies exactly the issue I consider most load-bearing: each model must produce a probability for every sample, even from other datasets with disjoint feature spaces, but the paper never specifies how. This is a correctness gap, not a stylistic or consensus disagreement. Without a defined scoring mechanism, Eq. (1) cannot be evaluated on the cross-domain validation set, so the main result (fused macro F1 = 0.823) is unsupported. I also note the additional validation-set leakage (grid search on the same set used for reporting), which would further undermine the claim even if the feature alignment were resolved. The paper provides no code or artifacts despite claiming full reproducibility, so an independent check is impossible. Given these issues, the verdict should remain REJECT. The reader's weakest_assumption is identical to my primary concern, so I agree with the reader's identification.","tokens_in":4742,"tokens_out":2638,"duration_ms":33400,"concrete_test":"Obtain the author's code/data (or request them). For one held-out EMBER sample, one held-out API sample, and one held-out CIC sample, call predict_proba on all three trained LightGBM models. If any model rejects the sample's feature vector (dimension mismatch) or requires an undocumented imputation/alignment step, the fusion equation cannot be evaluated as described. Then reproduce the unified validation set of 28,776 samples and check that each sample can be scored by all three models; if not, the reported 0.823 macro F1 is not reproducible.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim depends on Eq. (1) (Section III-C), which requires every validation sample to be scored by all three LightGBM models: y_fused = w1·y_ember + w2·y_api + w3·y_cic. However, the three datasets have disjoint feature spaces: EMBER uses 2,381 static features, API Calls uses 150 behavioral features, and CIC uses 200 memory features. After top-feature selection (top 50, 50, 20), the models' input dimensions remain incompatible. The text never specifies how a model trained on one feature set can generate a probability for a sample from another feature set—no feature alignment, imputation, common representation, or other bridging mechanism is described. The unified cross-domain validation set is pooled from held-out samples of all three datasets (28,776 samples), so for every sample in that set all three y_i values are needed to compute the fused score. Since the paper does not define how y_api and y_cic are produced for an EMBER sample (or vice versa), the reported macro F1 of 0.823 is not a well-defined quantity under the stated methodology. This is not merely a missing detail: it makes the central experiment physically impossible to execute as described unless an unstated assumption is introduced. Secondary to this, the same validation set is used for grid-search weight selection and for reporting the final F1, which would inflate the reported performance even if the scoring problem were solved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight cross-domain malware detection framework that trains separate LightGBM classifiers on three heterogeneous datasets—EMBER (static), API Call Sequences (behavioral), and CIC Obfuscated Memory (memory)—selects top features per dataset, and fuses the predicted probabilities with a weighted sum. Fusion weights are chosen by grid search on a pooled 'cross-domain validation set.' The authors report a fused macro F1 of 0.823, which they interpret as evidence of successful generalization across distinct feature spaces, and they claim full reproducibility. The core problem is that the fusion step is not well-defined: the three datasets have disjoint feature sets, and the manuscript never explains how a model trained on one feature set can produce a probability for a sample from another feature set. In addition, the same validation set is used both to select fusion weights and to report the final metric, making the headline result an in-sample fit. Several dataset split and sample-count inconsistencies further undermine the experimental claims.","tokens_in":5066,"tokens_out":5001,"duration_ms":59523,"significance":"If properly executed, probability-level fusion of lightweight gradient-boosting models is a practical and potentially valuable approach to combining complementary malware-detection signals, and cross-domain robustness is an important goal. The manuscript is clearly written, uses publicly available datasets, and states an intention to provide code and data. However, the central empirical claim—that the fused model generalizes better than any single-domain model—is not supported as presented. The undefined feature-space bridge and the circular validation make the reported 0.823 macro F1 non-reproducible and not a measure of generalization. These are fixable in principle with additional methodological detail and a properly separated evaluation, but they currently undermine the paper's central contribution. The paper also makes claims of 'full reproducibility' without providing a repository link.","major_comments":[{"comment":"Equation (1) requires every sample in the cross-domain validation set to be scored by all three models, but EMBER (2,381 features), API Calls (150 features), and CIC Memory (200 features) have disjoint feature spaces. After top-feature selection (50, 50, 20), the input dimensions are still incompatible. The text does not specify any feature alignment, common embedding, zero-padding, imputation, or other mechanism that would allow the API model to score an EMBER sample or the CIC model to score an API sample. Consequently, the fused probability and the headline macro F1 of 0.823 in Section IV-B are not well-defined quantities under the stated methodology; the experiment cannot be reproduced or executed as described.","section":"III-C, Eq. (1); IV-B"},{"comment":"The fusion weights are selected by an exhaustive grid search on the unified cross-domain validation set, and the same validation set is then used to report the fused model's macro F1-score. This is circular: the reported 0.823 is the result of fitting the weights to the evaluation set, not an independent measure of generalization. A proper evaluation requires a separate held-out test set or nested cross-validation for weight selection, and the final metric should be reported on data never used for weight selection. No confidence intervals or error bars are provided either.","section":"III-C; IV-B"},{"comment":"The validation set sizes in Table I are inconsistent with the stated 80/20 split. EMBER has 639,900 samples, so a 20% split should give 127,980 validation samples, not 63,990. API Calls has 35,100 samples, so a 20% split should give 7,020, not 11,720. CIC Memory is reported with 46,876 validation samples, which is the full dataset size and implies no held-out split. Furthermore, the combined cross-domain validation set is said to contain 28,776 samples, a number not derivable from any of these splits. These inconsistencies raise concerns about data leakage and make it impossible to interpret the individual model performances and the construction of the fusion validation set.","section":"III-A; Table I; IV-B"},{"comment":"The comparison in Table II is not apples-to-apples. The per-component macro F1 values (0.965, 0.867, 1.000) are the models' scores on their native validation sets, whereas the fused model's macro F1 (0.823) is computed on the pooled cross-domain validation set. The paper's claim that fusion 'significantly outperforms' individual models is therefore unsupported; it could be true that each individual model performs poorly on the cross-domain set while the fusion improves it, but that is not demonstrated. The table should report each model's macro F1 on the same cross-domain validation set used for the fusion.","section":"Table II; IV-B"}],"minor_comments":[{"comment":"Reference [2] is titled 'Microsoft Malware Classification Challenge,' but the API Call Sequences dataset is not clearly identified with this source. Reference [3] is described as the CIC Obfuscated Memory dataset, but the cited title is about system-call-based malware detection; please verify and correct the dataset citations.","section":"References [2], [3]"},{"comment":"The abstract and conclusion state that 'all code and data are provided for full reproducibility,' but no repository link, supplementary material, or data availability statement appears in the manuscript. This claim should be substantiated with a URL or appendix.","section":"Abstract; VI"},{"comment":"The composition of the 28,776-sample cross-domain validation set should be specified per dataset, along with label balance. The current description is too vague to assess whether the pooled set is representative.","section":"IV-B"},{"comment":"The text references subfigures 1a–1c and 2a–2c, but the figures as printed do not have explicit subfigure labels. Please add labels or revise the references.","section":"Figures 1 and 2"},{"comment":"There are minor formatting issues, such as an extra space in Reference [4] ('Y . Ke') and inconsistent citation formatting in Reference [8]. Please proofread references.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript should not be accepted in its current form. The undefined feature-space bridge and the circular evaluation are load-bearing flaws that make the reported results non-reproducible. However, the underlying idea is reasonable and the flaws are potentially addressable: the authors could clarify how cross-model scoring is performed (e.g., a common feature representation) and re-run the evaluation with a genuinely independent test set and corrected data splits. If the code is available, it should be checked to see whether the reported numbers are actually reproducible under a sensible interpretation. If the authors cannot provide a coherent explanation of how Eq. (1) is computed, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is a clear reject for me. The core experiment is undefined: Eq. (1) requires every validation sample to be scored by all three LightGBM models, but EMBER, API Calls, and CIC have disjoint feature spaces (2,381 vs. 150 vs. 200 features, then top-50/top-20). The paper never says how a model trained on one feature set produces a probability for a sample from another. No alignment, imputation, or common embedding is described. So the reported macro F1 of 0.823 on the pooled 28,776-sample set is not a well-defined quantity under the stated method. That alone is fatal.\n\nTo give credit: the writing is clear, the idea of fusing static, behavioral, and memory signals at probability level is sensible, and the paper correctly cites Kittler et al. for weighted classifier combination. The individual model results are plausible and presented with confusion matrices and calibration plots, which is more than many short papers do.\n\nBut the soft spots are large. First, the evaluation is circular: the same validation set is used both for grid-search weight selection and for reporting the final fused F1. The 0.823 number is therefore a fit, not a generalization measure. There is no separate test set and no error bars. Second, the claim of \"superior generalization\" is unsupported on its own terms: the fused F1 (0.823) is lower than every single model's native F1 (0.965, 0.867, 1.000). Since the validation set is just a pool of native-domain holdouts, the fused model is averaging high-confidence predictions on each model's home turf with arbitrary predictions on foreign turf. That is not cross-domain generalization. Third, the ablation claims (12–18% drops, <2% feature sensitivity, 8–15% weight sensitivity) are presented as numbers without any supporting table or figure. And despite the abstract promising code and data, the full text gives no link or repository.\n\nIn short, this is a routine application of a known fusion technique to a new trio of datasets, and the main experiment is not reproducible as written. The author is probably capable of fixing the feature-alignment problem and running a proper held-out evaluation, but the paper as submitted does not merit referee time. I would desk-reject it and invite a resubmission if the methodology is made coherent and the evaluation is redone with a clean train/validation/test split.","headline":"The paper's central cross-domain fusion result is not computable as described because the models operate on disjoint feature sets, and the reported F1 is a fit to the very validation set used to pick the fusion weights.","tokens_in":5517,"tokens_out":2332,"would_cite":false,"duration_ms":30997,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By fusing the predicted probabilities of three lightweight LightGBM models trained on static, behavioral, and memory malware datasets, this paper claims a cross-domain macro F1 of 0.823—better than any single-domain model.","keywords":["malware detection","probability-level fusion","LightGBM","ensemble learning","cross-domain generalization","static features","API call sequences","memory analysis"],"falsifier":"Run the published code on a held-out EMBER sample and pass it to the API-trained and CIC-trained models under the paper's preprocessing. If either model cannot produce a probability because its features are absent, Eq. (1) is undefined for cross-domain samples and the 0.823 score cannot be reproduced as stated. If the code does align features, then recomputing the fused macro F1 on a leave-one-dataset-out split would settle whether the gain is genuine transfer or an artifact of mixing validation sets.","tokens_in":4618,"feed_emoji":"🛡️","tokens_out":7168,"duration_ms":77947,"temperature":0.7,"pith_summary":"This paper claims that a malware detector can generalize across very different data sources without training one big model on all features. The proposed recipe is to train three lightweight LightGBM classifiers separately—one on static PE-header features (EMBER), one on API call sequences, one on memory-obfuscation patterns (CIC)—and then combine their predicted probabilities with weights chosen by grid search. The authors report a macro F1 of 0.823 on a validation set pooled from all three domains, where no individual model reaches that level outside its native domain. If the claim holds, cross-domain malware detection can be built from cheap specialists plus a weighted average, which is useful for real-time deployment and for datasets that cannot be concatenated because their feature spaces are incompatible.","feed_headline":"Probability fusion of three models hits 0.823 cross-domain F1","feed_subtitle":"A weighted average of static, API, and memory signals is cheap and generalizes better than any one dataset.","key_machinery":"The load-bearing object is Eq. (1), the weighted sum of three model probabilities, with nonnegative weights summing to 1 and optimized by grid search in 0.1 steps over a unified validation set. This fusion step is what turns three domain specialists into one cross-domain detector; it avoids feature concatenation and hard voting, and it is what the ablation studies perturb to show each domain's contribution.","core_discovery":"On the paper's own terms, the discovery is that the fused probability y_fused = 0.5·y_ember + 0.4·y_api + 0.1·y_cic, with weights found by exhaustive grid search, achieves macro F1 0.823 on a cross-domain validation set of 28,776 held-out samples. Each single model scores higher on its native set (EMBER 0.965, API 0.867, CIC 1.000), yet the fusion is presented as the detector that generalizes. The result is framed as an extension of ensemble learning: rather than merging features or voting hard labels, the framework preserves each model's confidence and lets the data decide the relative contribution of static, behavioral, and memory evidence.","pith_inferences":["A sharper test the paper does not run: train on two domains and test on the third in rotation; that would separate true zero-shot transfer from the advantage that comes from pooling in-domain validation sets.","Because the fused score is a convex combination, it cannot beat the best specialist on samples where all three probabilities agree; the practical gain must come from disagreement among the three models, so examining disagreement regions would explain where fusion earns its 0.823.","Grid search could be replaced by a logistic regression over the three probabilities; if that matches or exceeds 0.823 with fewer tuned weights, the specific weight values would matter less than the act of calibrating and combining probabilities.","The same probability-fusion recipe should transfer to other heterogeneous malware datasets, provided each model's probabilities are well-calibrated; the next natural check is whether weights estimated on one holdout set remain optimal on an independent collection."],"forward_implications":["If the central claim holds, a real-time endpoint could run three LightGBM models on the same sample and combine their scores with fixed weights, keeping inference cheap enough for deployment.","Static features carry the most transferable signal (weight 0.5), behavioral features add context (0.4), and memory patterns act as a specialist refinement (0.1).","Removing any one of the three datasets reduces macro F1 by 12–18%, so the three domains are not redundant.","The weights are not arbitrary: deviations from the optimal combination cost 8–15% macro F1, which justifies the grid-search step rather than simple averaging."],"supporting_citations":[{"why":"Supplies the EMBER static-feature dataset that trains the first specialist model and contributes weight 0.5.","marker":"[1]"},{"why":"Supplies the API call sequence dataset for the behavioral specialist (weight 0.4).","marker":"[2]"},{"why":"Cited as the source for the CIC Obfuscated Memory dataset used by the memory specialist (weight 0.1).","marker":"[3]"},{"why":"Provides the LightGBM gradient boosting algorithm used to train all three domain classifiers.","marker":"[4]"},{"why":"Supplies the theoretical basis for combining classifier outputs at the probability level rather than by voting or feature fusion.","marker":"[5]"},{"why":"Represents the feature- and decision-level fusion approaches that the paper positions its probability-level fusion against.","marker":"[6]"}],"fun_headline_variants":["Blend of 3 model probabilities hits 0.823 cross-domain F1","Weighted fusion of static, API, memory beats single models","Lightweight fusion: 0.823 F1 via probability averaging"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire method assumes every sample can be scored by all three models, even though EMBER, API-call, and CIC-memory datasets use disjoint feature sets; the paper never specifies how a model trained on one feature set produces a probability for a sample described only by the other features.","fun_headline_variants_meta":{"raw":{"variants":["Blend of 3 model probabilities hits 0.823 cross-domain F1","Weighted fusion of static, API, memory beats single models","Lightweight fusion: 0.823 F1 via probability averaging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2565,"prompt_tokens":698,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":442,"tokens_out":1867,"duration_ms":17265,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:31:25.063021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published code on a held-out EMBER sample and pass it to the API-trained and CIC-trained models under the paper's preprocessing. If either model cannot produce a probability because its features are absent, Eq. (1) is undefined for cross-domain samples and the 0.823 score cannot be reproduced as stated. If the code does align features, then recomputing the fused macro F1 on a leave-one-dataset-out split would settle whether the gain is genuine transfer or an artifact of mixing validation sets.","supporting_citations":[{"cited_title":"A Quantitative Study of Accuracy in System Call-Based Malware Detection,","cited_arxiv_id":null,"evidence_quote":"Cited as the source for the CIC Obfuscated Memory dataset used by the memory specialist (weight 0.1)."},{"cited_title":"LightGBM: A Highly Efficient Gradient Boosting Decision Tree,","cited_arxiv_id":null,"evidence_quote":"Provides the LightGBM gradient boosting algorithm used to train all three domain classifiers."},{"cited_title":"On Combining Classi- fiers,","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical basis for combining classifier outputs at the probability level rather than by voting or feature fusion."},{"cited_title":"Multi-source malware detection using feature and decision-level fusion,","cited_arxiv_id":null,"evidence_quote":"Represents the feature- and decision-level fusion approaches that the paper positions its probability-level fusion against."}],"review_version":1}