{"id":"709bd2db-4726-4fe7-8eb0-e232b2c85468","arxiv_id":"2607.24177","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Under a unified four-axis benchmark, EMBER feature-based GBDT outperforms end-to-end deep malware detectors on the joint tradeoff of accuracy, temporal stability, adversarial robustness, and inference cost.","lead":"EXE-Bench ranks 30 AI Windows malware detectors on accuracy, time drift, adversarial robustness, and inference speed under one fair setup. Feature-engineered models beat most deep networks once real deployment constraints are scored together.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The #1 ranking rests heavily on the temporal metric: Table I’s GBDT–BBDnn gap is tiny, while T is timestamp-proxied and sensitive to uneven, possibly artifactual bins, so plausible re-binning or re-weighting could reorder the top.","rationale":"I read the paper as a serious benchmark effort: matched EMBER training, explicit content-injection attacks, CPU inference timing, public code/dashboard, Friedman ranks, and LOO sensitivity are real strengths and deserve credit. My concern is not that the methodology is invalid; it is reusable even if this instantiation’s temporal labels are imperfect. Nor is this a consensus objection: it is an internal stability issue in the quantity that carries the headline. The reader’s weakest assumption points at the right place—PE timestamps as a temporal proxy—but I would sharpen the mechanism: even if timestamps are “best available,” Eq. 1’s sample-weighting plus severe bin imbalance can amplify timestamp artifacts into the single metric where GBDT most clearly separates from BBDnn. Since the published aggregate gap is only about 0.01–0.03 depending on rounding, the top claim should be treated as conditional on the temporal construction and score weights. I would not move the verdict beyond the reader’s CONDITIONAL unless the proposed re-binning/re-weighting test flips the top rank or collapses the T gap; in that case the paper remains useful as a benchmark, but the strong production guidance and “GBDT superior” framing would need weakening.","tokens_in":23037,"tokens_out":3550,"duration_ms":116631,"concrete_test":"Run one temporal-axis sensitivity package on the released artifacts: recompute T and S after (a) re-binning the VT-covered Speakeasy subset by first-seen/first-submission rather than PE timestamp, (b) equal-weighting bins instead of |Si|-weighting, and (c) leave-one-bin-out over the 10 quarters. If EmberGBDT remains #1 with the same sign T gap over BBDnn and rank correlation ρ>0.9 to Table I, the concern is not load-bearing; if BBDnn overtakes or the T gap collapses when dominant bins are removed/re-weighted, downgrade the headline to weighting-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim needs more than “all models saw the same data”; it needs the aggregate S in Eq. 5 to preserve a real ordering. The soft spot is T (Eq. 1) and its influence at the top. In Table I, EmberGBDT beats BBDnn only narrowly in S (0.86 vs 0.85). Component-wise, GBDT’s clear edge is T (0.78 vs 0.67); P and R are nearly tied (0.99/0.97, 0.80/0.79), while BBDnn wins I (0.98 vs 0.87). Thus the headline “feature engineering remains superior” is substantially carried by the temporal axis.\n\nThat axis is the least secured. §IV-A uses PE compilation timestamps after dropping past/future outliers, while acknowledging timestomping and that VT ground truth is sparse: ~33% VT presence, ~7% non-null first-seen, and 45% of first submissions after collection. Those checks do not rule out plausible forged or build-farm timestamps landing inside valid-looking quarters. Eq. 1 then weights each quarter by |Si|/|D| to avoid the AUC fallacy, but Fig. 3/Tab. IV indicate highly uneven bin masses; a few dominant or contaminated bins can therefore set T for every model. This is not merely external-validity grumbling: because the S margin is small, a modest T perturbation can flip #1/#2 and change the practical conclusion from “GBDT wins” to “GBDT and BBDnn are comparable; choice depends on deployment weights.” The paper’s LOO/dashboard mitigations help exploration, but the abstract/leaderboard still present a single ordering.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper introduces EXE-Bench, a benchmark that re-trains 30 AI-based static Windows malware detectors (feature-based GBDT, end-to-end byte CNNs, an image classifier, and randomized/de-randomized smoothing certifiable variants) on one retrieved EMBER-2017 split and ranks them by an equally-weighted average S=(P+T+R+I)/4 of: F1 at deployment (P, at 1% FPR where thresholds are tunable), a novel sample-cardinality-weighted F1 over 10 four-month temporal bins from the Speakeasy dataset (T), a normalized AUC of detection rate vs. Levenshtein perturbation budget over three black-box genetic attacks (R), and an exponential of CPU inference time (I). The headline finding is that EmberGBDT tops the leaderboard (S=0.86 vs BBDnn 0.85), supporting the claim that feature engineering still beats end-to-end deep learning for deployable detectors, while certifiable and image-based models fare poorly. Code and an interactive reweighting dashboard are released. Statistical support includes Kendall W=0.48 (p=0.0021) across metric rankings and a leave-one-out rank-correlation sensitivity analysis.","tokens_in":23442,"tokens_out":4613,"duration_ms":140859,"significance":"If the central ordering holds, this is a useful community artifact: a controlled, apples-to-apples comparison of 30 Windows malware detectors with released code and an interactive reweighting dashboard, a principled sample-weighted temporal metric, explicit attack budgets with security-evaluation curves, and a realistic CPU inference protocol. The finding that certifiable defenses collapse under multi-axis evaluation, and that image-based detectors are both brittle and drift-prone, is genuinely informative for deployment decisions. The paper is also unusually candid about its limitations (timestomping, missing attacks, truncated inputs for RS models). However, the marquee claim — GBDT as the single winner — currently rests on a 0.01 margin whose deciding component (T) is the least-validated axis, so the practical significance is closer to \"GBDT and BBDnn dominate, with the ordering dependent on deployment weights\" than the abstract suggests.","major_comments":[{"comment":"The headline result (\"feature engineering remains superior\", GBDT ranked #1) is load-bearing on the temporal metric and on equal weighting, and the margin is 0.01. From Table I: GBDT vs BBDnn is nearly tied on P (0.99/0.97) and R (0.80/0.79), BBDnn wins I (0.98/0.87), and GBDT's edge is entirely T (0.78/0.67). Simple arithmetic shows the #1 slot flips under plausible changes: averaging only P, R, I gives BBDnn 0.913 vs GBDT 0.887. Meanwhile T is the least-secured axis (forgeable PE timestamps, sparse VT validation per Tab. IV). The paper's own LOO analysis (Tab. V) does not test the top of the leaderboard, only global rank correlation. The authors should (i) report a sensitivity analysis of the top-3 ordering to re-binning (e.g., quarterly vs 4-month windows, different outlier thresholds), timestamp perturbation, and metric weight perturbations around the equal-weight point; and (ii) tem","section":"§V-A, Table I, Eq. (5)"},{"comment":"The temporal protocol removes an apparently very large fraction of Speakeasy as outliers (Fig. 3's \"Past Out.\" bar dwarfs the retained bins, on the order of half the 125,921 samples), and the retained bins are extremely uneven (roughly 192 to 18,864 samples per bin per class). Since T weights bins by |Si|/|D| (Eq. 1), a few dominant bins — whose timestamps are exactly the ones most likely to be build-farm artifacts or timestomped values that survived the outlier filter — effectively set T for every model. Table IV shows the VT cross-check cannot adjudicate this: 33% presence, ~7% non-null first-seen, and 45% of first submissions after collection. Please report exact discarded counts and per-bin class composition, and quantify T's stability (e.g., bootstrap over bins, or T recomputed excluding the two largest bins). Without this, the reader cannot tell whether the GBDT–BBDnn T gap (0.11)","section":"§IV-A, Fig. 3, Table IV"},{"comment":"R values in Table I mix two different evaluation regimes: non-certifiable models are attacked directly, while certifiable models are evaluated only via transfer attacks (§IV-C), because direct attacks were computationally infeasible. Presenting both in one column invites misreading (e.g., BBDnnRS R=0.62 ranking above directly-attacked models may reflect attack non-transfer rather than robustness; cf. BBDnnRS's odd P=0.67). Additionally: (a) the normalization of R by the global maximum ε across all models couples each model's score to the hardest attack run against any model — adding a model or attack to the benchmark changes existing R values, which should be stated; (b) GAMMA with λ=0 admits empty .rdata section injections counted toward the perturbation budget; (c) robustness is measured on only 500 unpacked EMBER test samples with a single 500-query GA budget and no white-box evaluati","section":"§III (Eqs. 2-3), §IV-C, §V-D"},{"comment":"All metrics are computed from a single random 80/10/10 split and a single training run per model, with no variance estimates anywhere (Figs. 5-7, Tables I-II). Neural network rankings near the top (e.g., MalConv vs NGramConv on P; the T gap itself) could plausibly be within seed variance. Given that the benchmark's purpose is a durable ordering, at least the key claims (GBDT vs BBDnn on T and R) should be supported by repeated runs or split-level confidence intervals. Relatedly, Kendall W=0.48 among four metric rankings is invoked to \"justify\" aggregation into S, but concordance among metrics does not justify equal weights — these are distinct statements; please rephrase or support the weighting choice directly.","section":"§IV-B, §V-A"}],"minor_comments":[{"comment":"The rows are not sorted by S: e.g., BBDnnRsDel (0.56) appears above MalConv (0.67), BBDnnRDRS/SDRS (0.68/0.69) above several 0.65-0.67 entries. Bold/tie marking is also inconsistent (e.g., I=0.99 ties across several models are variously bolded). Please re-sort and re-check.","section":"Table I"},{"comment":"Model naming oscillates between BBDNN (§IV-B) and BBDnn/BBDnn (tables/figures); also \"subject toconcept drift\" (missing space) in §I, and Table I has run-together values (\"0.520.99\"). Table IV's column alignment is hard to parse.","section":"Notation/typos"},{"comment":"Radar plots are nearly unreadable at this density; consider a per-metric heatmap or parallel-coordinates plot, and mark the top-3 models.","section":"Fig. 4"},{"comment":"I = exp(-t) squashes differences above ~1s (BBDnnRsDel I=0.05 corresponds to ~3s; the distinction between 3s and 30s is deployment-relevant but nearly invisible in I). The footnote mentions a linear variant; please report in an appendix whether the leaderboard changes under it.","section":"§III, Inference Metric"},{"comment":"The inference-timing sample is restricted to file sizes within µ±0.5σ of Speakeasy (§IV-D), which excludes large files where feature extraction and input-window differences matter most. A sentence on how results change with k (or on the full size distribution) would strengthen the I metric.","section":"§IV-D"},{"comment":"BBDnn's robustness is partly attributed to its 100KB input window letting injected content fall outside the model's view (§V-D) — this is an important caveat that deserves prominence in the main results, since it means R partly rewards input truncation rather than robustness.","section":"§V-D"},{"comment":"EMBER2024 [42] is cited as future work but, given the timestamp concerns with Speakeasy, a brief discussion of whether EMBER2024's temporal splits would give a better-validated T axis would be valuable.","section":"§VI"}],"recommendation":"major_revision","confidential_remarks":"A substantial fraction of the evaluated techniques (de-randomized/randomized smoothing variants, the Speakeasy dataset itself, maltorch) originate from the author team or their close collaborators, and the citation pattern leans heavily on the group's prior work. This does not invalidate the benchmark — retraining everything under one protocol arguably mitigates the conflict — but the editor may want to note that the \"fair comparison\" framing coexists with the authors evaluating their own prior art, and that the leaderboard's main loser (randomized smoothing) is partly an artifact of the evaluation regime (transfer-only attacks, no certified-accuracy metric, truncated inputs), which the authors do acknowledge."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a careful same-data leaderboard of 30 static Windows detectors across accuracy, drift, content-injection robustness, and CPU inference. The useful finding is that EMBER GBDT stays near the top on almost every axis while most end-to-end nets look strong only right after training, and image-style ResNet is weak under attack. That is worth having in one place.\n\nWhat is actually new is not any single ingredient (EMBER, MalConv-family nets, adversarial EXEmples, TESSERACT-style time splits, RobustBench-style multi-metric scoring all exist). It is the matched protocol at this scale, the sample-weighted temporal score that correctly criticizes plain AUC over uneven bins, public code plus a dashboard, and a concrete ranking people can argue with. Training, thresholds, attack budgets, and CPU timing are controlled enough that the internal comparisons are fair. Kendall W and LOO are reported; limitations are mostly stated in the open. Circularity is low: S is an explicit average of measured quantities.\n\nSoft spots, in proportion. The headline “feature engineering wins” is real on the joint score, but Table I shows EmberGBDT 0.86 vs BBDnn 0.85. P and R are nearly tied; BBDnn wins inference; GBDT’s clear edge is T (0.78 vs 0.67). T is built from PE timestamps after outlier drops, with sparse VT first-seen coverage on Speakeasy. Timestomping and uneven bin mass are real risks, so a modest re-binning or re-weight could flip #1/#2 without touching the broader story that pure byte CNNs and image folds degrade under time and injection. Equal weights in S, transfer-only attacks on certifiable models, and incomplete attack/model coverage are secondary and partly mitigated by the dashboard. None of that sinks the experiment; it means the abstract should not be read as a definitive production ranking without stress-testing T and the weights.\n\nWho it is for: people who ship or compare static AV ML, and anyone tired of incomparable malware papers. Math and citation pattern look fine for an empirical systems piece. I would send it to peer review, ask for clearer sensitivity on temporal labeling and weightings near the top, and keep the leaderboard as a living artifact rather than gospel. Engage with it; cite the protocol and the component tables, not only the single S ordering.","headline":"Solid multi-axis malware-detector benchmark; GBDT wins the joint score, but the #1 vs #2 gap is thin and mostly rides on a timestamp-proxied temporal metric.","tokens_in":24377,"tokens_out":604,"would_cite":true,"duration_ms":21058,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"When accuracy, time drift, attacks, and speed are scored together, hand-crafted features still beat most deep networks for real-world Windows malware detection.","keywords":["malware detection","benchmark","Windows PE","machine learning","deep learning","adversarial robustness","concept drift","inference cost"],"falsifier":"Replace PE timestamps with reliable first-seen dates for most Speakeasy samples, recompute the temporal metric on the same thirty models, and check whether the EMBER gradient-boosted tree still tops the four-metric leaderboard.","tokens_in":24012,"feed_emoji":"🛡️","tokens_out":916,"duration_ms":39306,"temperature":0.7,"pith_summary":"Existing comparisons of AI Windows malware detectors cannot decide what to deploy: they train on different data, ignore concept drift, skip systematic adversarial testing, and overlook inference cost on ordinary endpoints. EXE-Bench fixes that by training thirty models on identical data and ranking them with one average of four metrics—immediate F1, sample-weighted F1 on later time bins, robustness under growing content-injection budgets, and inverse inference time. Under this ranking the classic gradient-boosted tree on EMBER features comes first, while most end-to-end deep networks look strong only right after training and then fall behind. The result matters because it shows practitioners which detectors remain usable once time and attacks enter the picture, rather than which ones merely top a static test set.","feed_headline":"Feature engineering still wins Windows malware detection","feed_subtitle":"Four-metric benchmark ranks thirty detectors; most deep nets fade under time and attacks","key_machinery":"EXE-Bench score S(f)=(P+T+R+I)/4: P is F1 on the held-out test set, T is the sample-weighted average F1 across future four-month bins, R is the normalized area under the detection-rate-versus-perturbation-budget curve, and I=exp(−average inference time). This single scalar produces the leaderboard.","core_discovery":"When thirty AI-based Windows malware detectors are trained and tested under identical conditions and scored jointly on immediate performance, stability under temporal drift, resistance to adversarial content injection, and inference speed, the gradient-boosting model that uses hand-engineered EMBER features achieves the highest aggregate score. Most end-to-end deep networks excel mainly on the deployment-time test set and degrade under later data or under modest byte manipulations, while image-based and randomized-smoothing variants rank near the bottom.","pith_inferences":["The same multi-axis scoring template could be applied to Android or Linux detectors to test whether feature engineering retains the same edge outside PE files.","If PE-timestamp noise is the main bias, the relative ordering of feature-based versus end-to-end models may stay stable once cleaner time labels arrive, because both families see identical bins.","Certification schemes that slice files into chunks look more practical for endpoint latency than randomized smoothing, so research effort may pay off more by raising their clean accuracy than by adding further noise-based certificates.","The gap between immediate F1 and temporal/robustness scores implies continuous re-training will remain necessary for deep models even when feature models can be updated more cheaply."],"forward_implications":["Practitioners should prefer feature-engineered models over raw-byte deep networks when deployment must survive drift and content-injection attacks.","Image-based malware classifiers should be avoided in production under these four criteria.","De-randomized smoothing can improve speed and sometimes rank, but does not automatically confer robustness to section-injection attacks.","Accuracy-only leaderboards systematically over-rank models that overfit the training-time distribution.","New detectors can be dropped into the same four-metric average without redesigning the comparison."],"fun_headline_variants":["Hand-engineered features top 30 Windows malware detectors","EMBER gradient boosting beats deep nets on four-metric rank","Domain knowledge resists time and attacks in malware detection","Deep nets fade; feature engineering wins EXE-Bench ranking","Four-metric score crowns engineered features over end-to-end nets"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The ranking over time treats PE compilation timestamps (after discarding obvious outliers) as a good enough proxy for when each sample first appeared, even though those timestamps can be forged and external first-seen coverage is sparse.","fun_headline_variants_meta":{"raw":{"variants":["Hand-engineered features top 30 Windows malware detectors","EMBER gradient boosting beats deep nets on four-metric rank","Domain knowledge resists time and attacks in malware detection","Deep nets fade; feature engineering wins EXE-Bench ranking","Four-metric score crowns engineered features over end-to-end nets"]},"model":"grok-4.5","effort":"low","cost_usd":0.003315,"raw_usage":{"total_tokens":1143,"prompt_tokens":777,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":33148000,"prompt_tokens_details":{"text_tokens":777,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":297,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":777,"tokens_out":69,"duration_ms":6381,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T21:47:39.281713+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Replace PE timestamps with reliable first-seen dates for most Speakeasy samples, recompute the temporal metric on the same thirty models, and check whether the EMBER gradient-boosted tree still tops the four-metric leaderboard.","supporting_citations":[],"review_version":1}