{"id":"080af4af-4306-45df-ae65-ee87443a13a0","arxiv_id":"2607.09290","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Tsetlin Machine detects PDF malware at 98.02% accuracy with clause-based interpretability on the RIT-PDFMal-2026 static-feature dataset.","lead":"A Tsetlin Machine classifies PDF files as benign or malicious from static features, reaching 98.02% accuracy on the RIT-PDFMal-2026 dataset while exposing its decisions as logical clauses. Security teams may care because the model is fast, competitive with black-box ensembles, and intrinsically explainable without post-hoc tools.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Aggressive duplicate removal and undersampling may inflate accuracy and make the 98.02% claim non-transferable to real PDF streams.","rationale":"The Reader correctly isolates the weakest assumption: that the heavily cleaned and re-balanced feature matrix remains representative. My stress-test simply sharpens the same point into a concrete, falsifiable check (re-run without undersampling). No deeper internal inconsistency or mathematical error is present; the TM formulation, clause voting and interpretability visualizations are standard and correctly applied. The concern is therefore real but ordinary for applied malware papers, so the Reader’s CONDITIONAL verdict and HIGH confidence remain appropriate; no upgrade or downgrade is warranted. The proposed test would settle whether the reported numbers are robust or preprocessing artifacts.","tokens_in":10134,"tokens_out":588,"duration_ms":8137,"concrete_test":"Re-train and re-evaluate the identical TM configuration (250 clauses, T=15, s=5, 50 epochs, same binarizer) on the full post-duplicate-removal set without random undersampling, using stratified 5-fold CV or a class-weighted loss; report accuracy, macro-F1 and the malicious-class F1. If either metric drops by more than ~3–4 points relative to the published 98.02%/95.99%, the headline competitiveness claim is inflated by the preprocessing choices.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (98.02% accuracy / 95.99% macro F1, competitive with RF while intrinsically interpretable) rests on the RIT-PDFMal-2026 evaluation after discarding 8,966 duplicates (36.84% of the original 24,337 samples) and randomly undersampling the majority class only on the training split (Section V-C1, Figs. 3–4). After cleaning, only 2,222 malicious samples remain against 13,149 benign; the test set therefore contains a very small absolute number of malicious examples. The 42 static numerical features (Table I) are themselves highly structural (obj/endobj counts, JS/JavaScript flags, isEncrypted, etc.). If many of the discarded “duplicates” were near-duplicates that still carried distinct malware families or packing variants, or if the undersampled training distribution no longer matches the natural class prior and feature co-occurrence of live PDF traffic, both the accuracy number and the learned clauses/feature-contribution heatmaps (Figs. 7–15) become optimistic and non-transferable. The paper reports no ablation that restores the original class balance or re-inserts the near-duplicates, nor any external hold-out corpus, so the strongest empirical claim is only weakly supported by the single preprocessed split.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a Tsetlin Machine (TM) framework for static PDF malware detection. It extracts 42 numerical structural features without executing files, preprocesses the RIT-PDFMal-2026 corpus (duplicate removal, train-only undersampling, min-max scaling, KBinsDiscretizer binarization), and trains a TM with 250 clauses to classify benign vs. malicious PDFs. On a held-out 20% test split the model reports 98.02% accuracy and 95.99% macro F1, competitive with Random Forest (98.28%) and superior to XGBoost/LightGBM, while offering lower inference latency than RF and intrinsic interpretability via class-vote scores, clause-activation heatmaps, and feature-contribution plots. The authors position the combination of competitive accuracy, efficiency, and built-in explainability as the main contribution for practical PDF malware detection.","tokens_in":10433,"tokens_out":651,"duration_ms":5701,"significance":"If the reported accuracy and interpretability claims hold under realistic distributions, the work supplies a concrete, low-latency alternative to black-box ensembles for a high-volume attack vector. The TM’s propositional clauses and vote/feature visualizations are a genuine methodological advantage over post-hoc SHAP/LIME explanations commonly applied to PDF detectors. The experimental pipeline is transparent (stratified split, train-only undersampling, five-fold CV, class-wise metrics, confusion matrix, seven baselines). The main limitation is that all evidence rests on a single, heavily cleaned public corpus; external validation or multi-dataset results would be required before the practical-deployment claim can be considered established. Within those bounds the contribution is solid and of clear interest to the malware-detection community.","major_comments":[{"comment":"Section V-C1 and Tables II/IV: After discarding 8,966 duplicates (36.84% of the original 24,337 samples) only 2,222 malicious files remain. Random undersampling is then applied solely to the training split, so the absolute number of malicious test examples is small. No ablation restores the original class prior, re-inserts near-duplicates, or evaluates an external hold-out corpus. Consequently the headline 98.02% accuracy / 95.99% macro-F1 figures (and the learned clauses shown in Figs. 7–15) may be optimistic relative to live PDF streams; at minimum the paper should quantify sensitivity to these preprocessing choices or report results on an independent corpus.","section":null},{"comment":"Section V-D / Table VI: The state-of-the-art comparison juxtaposes methods evaluated on Contagio, Evasive-2022 and RIT-PDFMal-2026. Because the datasets differ in collection period, feature sets and class balance, the claim of “competitive performance … with existing methods” is only weakly supported. Either re-evaluate the baselines on the same RIT-PDFMal-2026 split or clearly qualify the comparison as non-head-to-head.","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean applied paper that puts Tsetlin Machines on PDF malware for the first time and ships the accompanying RIT-PDFMal-2026 corpus. That combination is the real novelty; neither the model nor static PDF features are new, but the concrete numbers, clause heatmaps, and public data are.\n\nWhat it does well is straightforward. After standard preprocessing they get 98.02% accuracy and 95.99% macro F1, within a hair of Random Forest and ahead of XGBoost/LightGBM, with faster inference than RF. The interpretability section is not hand-waving: class-vote bars, clause-activation heatmaps, and signed feature contributions are shown for both correct and incorrect cases. That is more transparent than the usual SHAP/LIME post-hoc layer. The experimental pipeline is also transparent—stratified split, undersampling only on train, five-fold CV for hyper-parameters, class-wise metrics, confusion matrix.\n\nThe soft spots are real but ordinary. Everything rests on one dataset after discarding 36.84% duplicates and undersampling the majority class; the remaining malicious count is modest, and there is no ablation that restores the original balance or tests an external corpus. So the 98% figure and the learned clauses may be optimistic for live traffic. No code release, no significance tests, no user study of the explanations. Those are engineering caveats, not load-bearing contradictions.\n\nMath and citations look fine: standard TM formulation, honest self-cites to the author’s radio-map work that do not prop up the malware claims, and a reasonable baseline set. Free parameters (clauses, T, s, bins) are reported.\n\nThis is useful for people who care about explainable static malware detectors and want a reproducible starting point. I would send it to peer review; a referee can demand the missing ablations and multi-corpus checks without the paper needing to be reinvented. Worth a look if you work in that niche; not required reading otherwise.","headline":"Solid first application of Tsetlin Machines to PDF malware: competitive accuracy, real interpretability demos, and a new public dataset, tempered by single-corpus evaluation after heavy cleaning.","tokens_in":11029,"tokens_out":514,"would_cite":false,"duration_ms":5362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A Tsetlin Machine detects PDF malware at 98 percent accuracy while explaining each decision through readable logical rules.","keywords":["Cybersecurity","Malware Detection","Portable Document Format","Tsetlin Machine","Interpretable Machine Learning","Static Analysis"],"falsifier":"Re-run the identical feature extraction, preprocessing, and TM training pipeline on a fresh, independently collected PDF corpus that keeps natural class balance and includes malware families arriving after the original collection window; a large drop below the reported 98 percent accuracy would falsify the generalization claim.","tokens_in":11016,"feed_emoji":"🛡️","tokens_out":823,"duration_ms":21316,"temperature":0.7,"pith_summary":"This paper sets out to show that PDF malware can be spotted by a Tsetlin Machine that learns human-readable logical clauses from static file features, without ever opening or running the document. On the RIT-PDFMal-2026 collection, after cleaning and balancing, the model reaches 98.02 percent accuracy and a macro F1 near 96 percent, staying competitive with Random Forest and gradient-boosting baselines while finishing inference in a few microseconds. Because every prediction is a transparent vote among propositional clauses, an analyst can see which rules and features drove the benign or malicious label. The work matters because PDFs remain a major attack channel and most high-performing detectors stay black boxes that need approximate post-hoc explanations. Intrinsic interpretability therefore turns competitive detection numbers into something security teams can audit and trust in practice.","feed_headline":"PDF malware detector hits 98% with readable rules","feed_subtitle":"Static features and logical clauses match black-box models while showing why each file was scored.","key_machinery":"The Tsetlin Machine: a rule-based learner that builds conjunctive clauses from binary literals and their negations, then aggregates signed clause votes (bounded by threshold T) to decide the class. Those clauses are the human-readable decision rules that make classification transparent.","core_discovery":"A Tsetlin Machine trained on 42 static numerical features extracted from PDF files without execution classifies them as benign or malicious at 98.02 percent accuracy (macro precision 96.03 percent, recall 95.95 percent, F1 95.99 percent) on the held-out RIT-PDFMal-2026 test set, matching or closely trailing strong black-box classifiers while supplying intrinsic explanations through clause activations, class-vote totals, and signed feature contributions.","pith_inferences":["Because the model relies only on static structural counts, it can serve as a cheap first-stage filter before slower sandboxes or dynamic analysis.","The higher false-negative rate relative to false positives suggests some malware families remain under-covered by the learned clauses; expanding the clause budget or feature set is a direct next experiment.","Readable rules open a path for human-in-the-loop editing in which analysts disable or refine clauses that fire on known false positives."],"forward_implications":["Security operators can inspect activated clauses and top feature contributions to audit why any given PDF was flagged or cleared.","Static analysis plus TM inference runs in roughly three microseconds per sample, supporting high-volume scanning.","Competitive accuracy is obtained without post-hoc explanation tools such as SHAP or LIME.","The same clause representation surfaces which structural PDF elements (JavaScript, Encrypt, Launch, XFA, etc.) dominate malicious patterns."],"fun_headline_variants":["Tsetlin Machine flags PDF malware at 98% via readable rules","Static features let interpretable TM hit 98% PDF malware detection","Rule-based Tsetlin classifier matches black-box PDF malware accuracy","Transparent TM scores PDF files malicious at 98.02% with clause votes","Interpretable Tsetlin framework detects PDF threats without execution"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The 42 static numerical features, after discarding more than a third of samples as duplicates and undersampling the benign class, still capture the patterns that matter for real-world PDF malware.","fun_headline_variants_meta":{"raw":{"variants":["Tsetlin Machine flags PDF malware at 98% via readable rules","Static features let interpretable TM hit 98% PDF malware detection","Rule-based Tsetlin classifier matches black-box PDF malware accuracy","Transparent TM scores PDF files malicious at 98.02% with clause votes","Interpretable Tsetlin framework detects PDF threats without execution"]},"model":"grok-4.5","effort":"low","cost_usd":0.00388,"raw_usage":{"total_tokens":1208,"prompt_tokens":746,"num_sources_used":0,"completion_tokens":96,"cost_in_usd_ticks":38800000,"prompt_tokens_details":{"text_tokens":746,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":366,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":746,"tokens_out":96,"duration_ms":8442,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T04:06:46.228124+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the identical feature extraction, preprocessing, and TM training pipeline on a fresh, independently collected PDF corpus that keeps natural class balance and includes malware families arriving after the original collection window; a large drop below the reported 98 percent accuracy would falsify the generalization claim.","supporting_citations":[],"review_version":1}