{"id":"1555d2ec-6475-42a5-b8c8-fed92620b9c2","arxiv_id":"2504.14886","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Alpha claims near-perfect zero-day malware detection by feeding assembly instructions from a dynamic instrumentation tool into a DistilBERT Transformer, but its evaluation fits key classifiers to the test set and relies on training-data overlap.","lead":"Alpha is a malware detector that combines a dynamic binary instrumentation tool with a Transformer model and two SVM layers to classify assembly instructions as malicious or benign. The paper reports perfect accuracy on ransomware, worms, and APTs using one minute of execution data, but the evaluation has design flaws that make the headline claims unsupported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Alpha's perfect zero-day scores for Ransomware/Worms/APT are largely produced by Layer 1, which classifies from counts of functions already seen in training, not from novel ASM functions; the zero-day claim is not established.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing flaw: Layer 1 resolves most of the perfect-score families by counting functions that match the training set, so the system is not evaluating novel malware behavior for those samples. My independent reading of Algorithm 1 and Table 13 confirms that 68 of 88 Ransomware, 21 of 26 Worms, and 10 of 14 APT malicious samples never reach DistilBERT, invalidating the zero-day interpretation of the 100% scores. I also note the additional protocol issues the reader enumerates—Layer 3 SVM trained on test samples, Layer 1 thresholds derived from test data, minute-3 selected because it gave perfect results, and inconsistent sample counts such as Worm total 26 in Table 13 versus 29 in Table 10—but the Layer 1 overlap problem alone is sufficient to reject the central claim. The dataset and architecture may have some value, but the evaluation protocol does not support the paper's headline conclusion. No change to the reader's REJECT verdict is needed.","tokens_in":20156,"tokens_out":3212,"duration_ms":30034,"concrete_test":"Rerun the Ransomware, Worm, and APT evaluations with Layer 1 disabled: every test sample is classified only by DistilBERT on the post-filter novel functions, with the Layer 3 SVM hyperplane and all thresholds fitted on the training split (or a held-out validation split) rather than on the test set. Report the metrics for these families. If Ransomware/Worms/APT no longer reach 100%, or if the 88/14/26 malicious sample counts cannot be reproduced, the perfect headline scores depend on Layer 1 overlap and/or test-set leakage, and the zero-day claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Alpha's headline result—100% accuracy for Ransomware, Worms, and APT—is reported for the full three-layer pipeline. But Algorithm 1 and Table 13 show that Layer 1 decides most of those samples using only a literal lookup of whether each function appears in the training set: for Ransomware 68/88 malicious samples, APT 10/14, and Worms 21/26 are resolved at Layer 1 by an SVM over (malicious-count, benign-count), without DistilBERT ever seeing a novel function. Layer 1 is therefore a training-overlap/memorization classifier, not a zero-day detector. The paper's own premise is that the model should classify 'truly new' functions after filtering out all functions seen in training; the perfect scores for these families largely measure re-identification of known functions, not generalization to unseen behavior. Compounding this, the Layer 1 thresholds are 'quartile-based metrics derived from the benign and malicious classifications' (Section 5.4), and Section 5.2 says the Layer 3 SVM is trained 'on a the test samples.' So the remaining DistilBERT stage also operates under test-set leakage. The central claim—detecting genuinely new malware samples with perfect accuracy—is therefore unsupported: no experiment isolates performance on novel functions for exactly the families that claim 100%.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Alpha, a three-layer malware detector that operates on Assembly-language function traces collected by the Peekaboo DBI tool. Layer 1 counts how many functions of a test sample match malicious/benign functions in the training set and uses an SVM to classify high-confidence samples; Layer 2 fine-tunes per-malware-type DistilBERT models to classify filtered 'novel' functions; Layer 3 applies an SVM to the fraction of functions labeled malicious. Experiments A, B, and C evaluate the full traces, one-minute slices, and the complete Alpha pipeline, respectively. The paper reports perfect accuracy for Ransomware, Worms, and APTs and high accuracy for other families, claiming zero-day detection by removing any test functions seen in training.","tokens_in":20452,"tokens_out":6690,"duration_ms":58076,"significance":"If the results held, the contribution would be significant: combining dynamically instrumented ASM traces with a Transformer model to classify novel malware within one minute, and releasing code and models, would advance zero-day endpoint detection. The paper also usefully extends the authors' prior Pulse work to multiple malware families and analyzes per-minute instruction density. However, the reported metrics are compromised by test-set usage in several places, and the zero-day claim conflates re-identification of known functions with generalization to novel ones. The strength of the contribution therefore depends entirely on a corrected evaluation.","major_comments":[{"comment":"The Layer 3 final classification SVM is stated to 'train on a the test samples with two features: Functions and Malicious %'. This SVM is used in Experiment A and again as Layer 3 in Experiment C and Algorithm 1, so the reported accuracy, precision, recall, and F1 for every family are in-sample numbers. A decision boundary fitted to the test samples cannot support claims of generalization to zero-day samples. The SVM must be fitted only on training data, or on a validation split, and then applied to a genuinely held-out test set.","section":"Section 5.2, Layer 3 SVM"},{"comment":"The Layer 1 thresholds are 'quartile-based metrics derived from the benign and malicious classifications'; because no validation split is described, these thresholds appear to be derived from the test samples themselves. Table 13 shows that Layer 1 decides most of the samples that produce the perfect scores (68/88 ransomware, 21/26 worms, 10/14 APT). Thresholds chosen from test-set distances make the reported 100% accuracy circular and must be set on training/validation data before evaluating the test set.","section":"Section 5.4, Algorithm 1"},{"comment":"Maze ransomware is used to select the one-minute slice, and after observing that minute 3 is the only minute with 100% accuracy, the paper uses minute 3 for all families in Experiment C. Selecting the time slice based on its test-set performance invalidates the subsequent Table 13 results as unbiased estimates. The choice of 'minute 2 to 3' in Experiment C versus 'minute 3' in Experiment B also needs clarification. A pre-registered slice choice based on training/validation data is required.","section":"Section 5.3, Table 11"},{"comment":"The central zero-day claim is not supported by the architecture as presented. The paper removes functions seen in training so that DistilBERT sees only novel functions, but Layer 1 classifies samples by counting functions that match the training set. Table 13 and Algorithm 1 show that for the families with perfect accuracy most malicious samples are resolved by Layer 1 (Ransomware 68/88, Worms 21/26, APT 10/14). This is overlap-based classification, not detection of 'truly new' functions. The authors should either restrict the zero-day claim to samples classified by Layers 2/3 after removing all overlapping functions, or report Layer 1 and Layer 2/3 results separately.","section":"Section 4 / Algorithm 1 / Table 13"}],"minor_comments":[{"comment":"Experiment B states that minute 3 is used, while Experiment C refers to 'a 1 minute data slice from minute 2 to 3'; the exact time interval needs to be specified consistently.","section":"Section 5.3 vs. Section 5.4"},{"comment":"The text claims that Alpha achieves 'perfect scores for Ransomware, Worms, APT, and Tools', but Table 13 reports Tools accuracy as 98.67% with one false positive; the discrepancy should be corrected.","section":"Section 6"},{"comment":"There are several typos that should be fixed in revision: 'Trasnformer' in Section 4, 'availabel' in Section 2, 'insturuction' in Section 6, 'a the test samples' in Section 5.2, 'Spetember' in reference [4], and 'independant' in reference [5].","section":"Throughout"},{"comment":"The column headers L3 TP/FN/FP/TN cover only flagged samples; the table should clarify how the Layer 1 counts are aggregated into the reported accuracy, precision, recall, and F1 columns.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for publication in its current form because the evaluation protocol uses test data for SVM fitting, threshold selection, and slice selection. I recommended major revision because a corrected evaluation is in principle possible with the released code and data, but if the corrected numbers no longer support the perfect-accuracy zero-day claim, the paper's main contribution should be reframed accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth knowing about for its Peekaboo DBI corpus and its one-minute-slice idea, but the headline zero-day results do not hold up. The evaluation leaks the test set in at least two places, and the Layer 1 mechanism that decides most samples is a training-overlap classifier, not a detector of novel functions.\n\nWhat is genuinely new: Alpha extends the authors' earlier Pulse ransomware work to seven malware types, adds a function-loss SVM layer, and tests one-minute slices of execution data. That is a legitimate engineering extension. They also release fine-tuned models and scripts, which is good practice and should be credited. The Zipf-law motivation for treating ASM like natural language is reasonable, and the architecture is not absurd.\n\nThe soft spots are load-bearing. Section 5.2 says the Layer 3 SVM is trained on the test samples, using Functions and Malicious Percentage as features. Section 5.4 says the Layer 1 thresholds are quartile-based metrics derived from the benign and malicious classifications, i.e., from test predictions. The one-minute slice was chosen after minute 3 gave perfect Maze results. Then Algorithm 1 and Table 13 show that Layer 1 resolves 68/88 ransomware, 21/26 worms, and 10/14 APT samples using only counts of functions already present in training. For those samples DistilBERT never sees a novel function. So the perfect scores for those families mostly measure re-identification of known functions and test-set-fitted thresholds, not generalization to unseen behavior.\n\nOne correction to the reader's report: I read Table 6 as putting Mimikatz in the training set, not the test set. The Tool test set is Backdoor.TeamViewer. That specific complaint is wrong, but it is minor next to the leakage problems.\n\nThere are also smaller presentation issues, like the odd Trojan row in Table 8, but I would not dwell on those.\n\nWho is this for? A researcher working on DBI-based feature extraction or ASM-level malware classification could get value from the dataset and the released models. A practitioner wanting a deployable zero-day detector should not trust the reported numbers. The architecture might be salvageable with a clean protocol: fit Layer 1 and Layer 3 on training data only, choose the time slice on a validation set, and report results separately for samples whose functions are truly novel versus those with training overlap.\n\nRecommendation: send it to peer review rather than desk-reject. The empirical core is substantive enough to warrant referee time, but it needs major revision before the zero-day claim is credible.","headline":"Alpha has a useful multi-type ASM dataset and a plausible fast-detection idea, but the perfect zero-day scores are produced by components fitted on the test set and by a Layer 1 overlap classifier, so the central claim is not supported as written.","tokens_in":20964,"tokens_out":2285,"would_cite":false,"duration_ms":23908,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Alpha classifies previously unseen ransomware, worms, and APTs with perfect accuracy using one minute of dynamic assembly-language execution traces.","keywords":["dynamic binary instrumentation","malware analysis","feature extraction","ransomware","transformers","LLM","AI","Assembly"],"falsifier":"Re-run Alpha on the same one-minute traces with Layer 1 bypassed and report accuracy only for samples whose functions have zero overlap with the training corpus; if accuracy drops far below 100% on ransomware, worms, and APTs, the perfect scores are explained by the overlap signal rather than by the Transformer's ability to classify novel functions.","tokens_in":19965,"feed_emoji":"🛡️","tokens_out":7589,"duration_ms":65759,"temperature":0.7,"pith_summary":"This paper claims that a framework called Alpha can classify malware it has never seen before—zero-day samples—by reading one minute of a program's assembly-language instructions captured while it runs. The authors argue that dynamic binary instrumentation defeats evasion and that the captured instruction stream, treated like natural language, carries enough signal for a Transformer model. Alpha removes every function from a test sample that appeared in training, so the remaining functions are supposedly novel, and then classifies the sample through a three-layer pipeline of two SVMs around a fine-tuned DistilBERT model. If the claim holds, endpoint defenses could flag new ransomware, worms, and advanced persistent threats in about a minute without waiting for signatures. The authors report perfect accuracy for Ransomware, Worms, and APTs, and above 96% for Trojans, Spyware, Botnets, and Tools.","feed_headline":"One-minute traces classify unseen ransomware, worms, APTs perfectly","feed_subtitle":"Alpha strips away known functions and still tells novel malware from benign software in about a minute.","key_machinery":"The central object is the 'function-loss' classification pipeline built on normalized assembly functions. Assembly instructions are concatenated with their operands, hexadecimal addresses are replaced by the placeholder memoryaddress, each function becomes a sentence, and custom WordPiece tokenizers map them for DistilBERT, a compact version of the BERT Transformer language model. A first SVM classifies a sample from the counts of removed functions that matched malicious versus benign training entries; only uncertain samples reach the Transformer, whose per-function predictions feed a final SVM that thresholds the malicious percentage. Zipf's law, a power-law frequency distribution common in natural language, is the stated reason Transformers should work on this material.","core_discovery":"On its own terms, the paper's central discovery is that a Transformer trained on normalized assembly instructions from a dynamic binary instrumentation tool can separate malicious from benign samples even after all functions shared with the training set are stripped away. Alpha's Layer 1 counts how many stripped functions were labeled malicious versus benign in training and lets an SVM decide with high confidence; the paper reports that this layer resolves most samples—68 of 88 ransomware and 10 of 14 APT samples—while uncertain samples pass to a DistilBERT model fine-tuned per malware type, and a final SVM thresholds the proportion of functions judged malicious. With one-minute slices of execution data, the paper reports 100% accuracy, precision, recall, and F1 for ransomware, worms, and APTs, with high but imperfect scores elsewhere. The paper reads this as evidence that the model generalizes to genuinely new instruction combinations rather than memorizing training functions.","pith_inferences":["The perfect ransomware, worm, and APT scores are driven largely by Layer 1 (68 of 88 ransomware and 10 of 14 APT samples), meaning the decisive signal for most samples is reuse of functions already seen in training rather than DistilBERT's judgment of novel code.","A trivial baseline that classifies by the ratio of matched malicious-to-benign functions would likely capture much of this signal; adding such a baseline would isolate the Transformer's contribution to genuinely novel functions.","The current evaluation already uses disjoint families in training and testing, but function overlap across families means sample novelty is not the same as function novelty; reporting results only on samples with zero overlapping functions would test the strongest zero-day claim.","A unified binary classifier without a known malware type, flagged as future work, would remove the per-type models' strong prior and is likely to reduce the reported scores."],"forward_implications":["A one-minute dynamic trace can replace the 10–15 minute runs used to build the dataset, making quick endpoint-style detection plausible.","Samples whose functions overlap with known malware can be flagged almost immediately by Layer 1, requiring no Transformer inference.","The reported perfect scores are specific to malware types that reuse functions across samples, while Trojans and Spyware still produce false positives and false negatives.","Because none of the ransomware samples in the test set encrypted within the first three minutes of execution, early detection would in principle precede the main damage."],"supporting_citations":[{"why":"Prior Pulse framework supplies the DistilBERT-on-ASM method and the function-removal strategy that Alpha extends from ransomware to all malware types.","marker":"[14]"},{"why":"Peekaboo DBI tool and dataset provide the evasive-behavior assembly traces every experiment is built on.","marker":"[2]"},{"why":"DistilBERT is the Transformer architecture fine-tuned for per-function classification in Layer 2.","marker":"[29]"},{"why":"DeepSemantic's instruction normalization informs the ASM preprocessing used to build functions as sentences.","marker":"[26]"},{"why":"Zipf's law justification for treating assembly instructions like natural-language words motivates the Transformer approach.","marker":"[25]"},{"why":"BERT pre-training and fine-tuning framework that DistilBERT inherits for the classification head.","marker":"[28]"}],"fun_headline_variants":["Perfect zero-day detection for ransomware, worms, APTs in 1 minute","Alpha removes known functions, still catches novel malware perfectly","Transformer on assembly traces: perfect classification of unseen malware","One-minute DBI traces: Alpha flags new malware with perfect accuracy","Stripping familiar code, Alpha IDs new ransomware, worms, APTs flawlessly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-day claim rests on treating functions that reappear from the training set as a legitimate detection signal: most perfect ransomware and APT results come from Layer 1 counting those overlaps, and if overlapping known functions does not count as detecting novel malware, the headline results collapse to a much smaller flagged subset.","fun_headline_variants_meta":{"raw":{"variants":["Perfect zero-day detection for ransomware, worms, APTs in 1 minute","Alpha removes known functions, still catches novel malware perfectly","Transformer on assembly traces: perfect classification of unseen malware","One-minute DBI traces: Alpha flags new malware with perfect accuracy","Stripping familiar code, Alpha IDs new ransomware, worms, APTs flawlessly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":4035,"prompt_tokens":956,"completion_tokens":3079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2988}},"tokens_in":572,"tokens_out":3079,"duration_ms":22214,"temperature":1.0,"reasoning_tokens":2988,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:37:49.701383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Alpha on the same one-minute traces with Layer 1 bypassed and report accuracy only for samples whose functions have zero overlap with the training corpus; if accuracy drops far below 100% on ransomware, worms, and APTs, the perfect scores are explained by the overlap signal rather than by the Transformer's ability to classify novel functions.","supporting_citations":[{"cited_title":"Gaber, M","cited_arxiv_id":null,"evidence_quote":"Peekaboo DBI tool and dataset provide the evasive-behavior assembly traces every experiment is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DistilBERT is the Transformer architecture fine-tuned for per-function classification in Layer 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepSemantic's instruction normalization informs the ASM preprocessing used to build functions as sentences."},{"cited_title":"Thurner, R","cited_arxiv_id":null,"evidence_quote":"Zipf's law justification for treating assembly instructions like natural-language words motivates the Transformer approach."}],"review_version":1}