{"id":"27c82b7d-e6ea-428f-9131-e6146331a1da","arxiv_id":"2502.07492","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RoMA combines learned global perturbations with contrastive and distribution regularization to train malware attribution models that stay about 80% accurate under PGD attacks.","lead":"This paper introduces a training method that helps AI models keep identifying which hacking group wrote a malware file, even when the file has been deliberately modified. It reports more than 80% accuracy under strong attacks, over twice the best comparison method, while training in about half the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness is measured under the same four-byte-region mask used in training, so the 80% PGD accuracy may not hold under the paper's own functionality-preserving attacker model.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: the robustness result depends on the set of bytes an attacker is allowed to modify. My stress-test sharpens this into a concrete evaluation-protocol concern. The paper's Section 3 threat model is broad: an attacker may use any functionality-preserving byte modification. But the method's adversarial training and, apparently, the evaluation attacks operate only on a fixed, small set of four byte regions. This is not merely a generalization worry; it is a mismatch between the claim and the stated threat model. If the PGD attack in Table 1 is generated with the same mask as Algorithm 1, then the comparison is fair among methods but does not measure robustness against the full attack surface the paper claims to address. A narrower attack mask can inflate all adversarial-training results, and the large margin over PGD-4-AT may be an artifact of the mask rather than of RoMA's global perturbation or consistency regularization. The proposed concrete test would settle this by expanding the mask to other known functionality-preserving byte regions and re-measuring Robust Accuracy. If the accuracy collapses, the central claim is over-scoped; if it holds, the concern is answered. The reader's CONDITIONAL verdict remains appropriate, as the paper should either demonstrate robustness under a broader attack surface or explicitly re-scope the claim. No other issue is more load-bearing: the lack of error bars is secondary given the size of the reported gap, and the C&W results are not the basis of the headline claim. I therefore recommend no change to the reader's verdict.","tokens_in":13444,"tokens_out":6018,"duration_ms":64473,"concrete_test":"Re-run the PGD-50 evaluation with an expanded byte-level attack mask that covers all functionality-preserving editable regions used in prior PE manipulation work (e.g., unused section-header fields, optional-header padding, import/export name padding, resource data, and padding beyond the current 100KB cap), while keeping the same epsilon, iteration count, and white-box setting, and verify with a functionality-preservation check. If RoMA's Robust Accuracy remains near 80% under this wider mask, the concern is resolved; if it drops toward the 30-40% range of PGD-4-AT, the headline robustness claim must be re-scoped to the four-position mask and the abstract's 'advanced adversarial attacks' claim should be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RoMA provides robust APT malware attribution against advanced adversarial attacks. Section 3's Attack Assumption states that the attacker can modify raw bytes through functionality-preserving transformations, with no restriction to specific byte regions. However, Section 4.2 and Algorithm 1 restrict adversarial generation to four perturbation positions: parts of the DOS header, a 1KB shift before the first section, slack space, and up to 100KB of trailing padding. Section 5.2's attack setting reports only epsilon and iteration counts for PGD; it does not state whether the evaluation attack uses the same four-position mask. If it does, the reported Robust Accuracy of about 80% is an accuracy against attacks confined to the exact regions RoMA was trained to defend, not against the broader class of functionality-preserving manipulations allowed by the paper's own threat model. PE files contain other editable, functionality-preserving byte regions (e.g., unused fields in section headers, optional-header padding, import/export name padding, resource data), and the paper provides no evidence that robustness transfers to those regions. Because the comparison methods are evaluated under the same restricted attack, RoMA's large margin over PGD-4-AT is consistent with the attack mask being a favorable evaluation condition rather than with genuinely superior robustness under the stated attacker model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RoMA, a single-step adversarial training method for byte-level APT malware attribution. RoMA combines a Global Perturbation (GP) pool, which stores learned perturbation patterns in embedding space and is selected per-sample via a contrastively trained Selection Head, with two consistency regularizers (Adversarial Contrastive Loss and Adversarial Distribution Loss) that encourage the model to treat clean and adversarial samples consistently. The authors introduce a new APT dataset AMG18 (6360 samples, 18 groups) and report experiments comparing RoMA against seven baselines, including non-adversarial MalConv/AvastNet, FGSM-AT, PGD-2-AT, PGD-4-AT, and image-domain adversarial training methods (FGSM-RS, NuAT). The main reported results are that RoMA achieves about 80% Robust Accuracy under PGD-50/60/70 while the next-best method (PGD-4-AT) reaches about 36%, and that RoMA trains in about half the time of PGD-4-AT, while also achieving the highest clean accuracy (91.00%).","tokens_in":13722,"tokens_out":7330,"duration_ms":67392,"significance":"If the claims hold, the paper would be the first systematic study of adversarial training for APT malware attribution at the byte level, and it would demonstrate a substantial robustness improvement over existing approaches. The GP mechanism, which explicitly learns shared perturbation patterns and uses them to generate stronger adversarial examples, is a novel idea, and the consistency regularizers are well-motivated for the attribution task. The public release of the trained model and dataset is a positive element for reproducibility. However, the significance is tempered by a missing alignment between the stated threat model and the evaluation attack surface, and by the absence of statistical reliability measures; these issues must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The threat model in Section 3 states that the attacker can modify raw bytes through functionality-preserving transformations with no restriction on which bytes are modified, yet both the training procedure (Algorithm 1) and the reported evaluation appear to confine perturbations to four predefined PE regions: the DOS header (excluding MZ and the PE pointer), a 1KB shifting space before the first section, slack space between sections, and up to 100KB of trailing padding. The paper does not explicitly state in Section 5.2 whether the PGD and C&W attacks used for evaluation apply the same four-position mask. If they do, the claimed ~80% Robust Accuracy is measured only against attacks restricted to the exact regions RoMA was trained to defend, which is not equivalent to robustness under the broad threat model stated in Section 3. Other editable, functionality-preserving byte regions (e.g., unused section-header fields, optional-header padding, import/export name padding, resource data) are not tested, so no evidence is provided that robustness transfers to those regions. The authors should either narrow the threat model to the four-position set with an explicit justification that these positions constitute the realistic attack surface, or broaden the evaluation to additional editable regions and demonstrate that RoMA's robustness advantage persists.","section":"Section 3, Section 4.2, Section 5.2"},{"comment":"All reported accuracy numbers (SA, RA, ASR) are single runs without error bars, multiple seeds, or significance tests. The claim of 'superior standard accuracy' rests on a 91.00% versus 90.06% gap over MalConv, which is likely within run-to-run variability for this type of model; similarly, the training-time comparison (2378 vs 4965 minutes) is a single observation. The authors should report the mean and standard deviation over at least three independent training runs for each method, and include a significance test (e.g., paired bootstrap or Wilcoxon test) for the key SA and RA comparisons.","section":"Table 1, Section 5.2"},{"comment":"The hyperparameters K=50, λ1=λ2=0.3, τ=0.6, and ε=0.6 are described as 'selected through extensive tuning,' but no validation split or tuning protocol is given. If the test set was used to select these values, then the comparison against baselines is biased, as the baselines do not appear to have received comparable tuning. The authors should specify the validation split used for model selection, report the ranges over which hyperparameters were tuned, and state whether the baselines' hyperparameters were tuned on the same validation set.","section":"Section 5.2"},{"comment":"The C&W attack evaluation is underspecified: the paper reports only that C&W is an optimization-based attack and gives a single row of RA/ASR numbers, with no details on the attack configuration (e.g., learning rate, number of iterations, binary-search steps, confidence parameter). Given that the contributions explicitly claim robustness against 'optimization-based attacks,' the paper should provide the full C&W attack settings and ideally evaluate over a range of attack strengths or budgets, rather than a single configuration that shows all adversarial-training methods performing similarly.","section":"Section 5.2, Table 1"}],"minor_comments":[{"comment":"The text refers to 'Table A3' for the experiment results, but the displayed main-results table is numbered 'Table 1,' while the appendix also contains a dataset table numbered 'Table A3.' This duplicate reference should be resolved by renumbering the main table and updating all cross-references.","section":"Sections 5.3–5.4"},{"comment":"The phrase 'more than twice as fast as the second-best method in terms of accuracy' is confusing; the intended comparison is training time, not accuracy. Please rephrase to say 'more than twice as fast in training time' or similar.","section":"Abstract"},{"comment":"The adaptations of FGSM-RS and NuAT from the image domain to malware attribution are not described. Provide implementation details, including how the perturbations are projected onto the four PE positions, to allow reproducibility and fair comparison.","section":"Section 5.2"},{"comment":"The provenance of the newly added samples in AMG18 is described only briefly as 'hash-label pairs from public threat repositories.' A more detailed data-availability statement, including exact source repositories and sample hashes, would strengthen the dataset's usability and credibility.","section":"Section 5.1"},{"comment":"The sentence 'The RoMA-trained model and visualized malware samples are publicly available' is odd; presumably 'visualized representations' is intended. Please clarify.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the proposed method shows large improvements in the reported setting. The main risk is the mismatch between the stated threat model and the evaluation attack mask, which could change the interpretation of the central claim. I recommend the editor require the authors to clarify and, if necessary, expand the attack evaluation, and to add statistical reliability measures. The dataset provenance and the reported C&W configuration should also be scrutinized carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: this is a real empirical contribution to a thin subfield—robust APT malware attribution at byte level—and the reported gap over baselines is large enough that the method deserves a serious look. But the headline number is only as good as the attack mask used to measure it, and the paper does not tell you whether that mask matches the attacker model it states. So treat the 80% figure as “robust under their four-region perturbation schedule,” not as robust against arbitrary functionality-preserving edits.\n\nWhat’s new: first to apply adversarial training specifically to malware attribution (not detection) with a learned global perturbation pool, a selection head, and two consistency losses. The combination is new relative to the cited literature. The AMG18 dataset, while an extension of APTMalware with added samples and labels, is a reasonable contribution. The ablation is internally consistent: removing GP crashes robustness, ACLoss helps clean accuracy, ADLoss gives a smaller boost. The t-SNE visualizations support the qualitative story.\n\nWhere it gets soft. The attack assumption in Section 3 says the attacker can make any functionality-preserving byte modification, but Algorithm 1 and Appendix A restrict training perturbations to four specific regions: DOS header, 1KB shift, slack, padding. Section 5.2 gives epsilon and iterations for PGD but never says whether the evaluation attack uses the same four-region mask. If it does—and the implementation detail that FGSM-AT uses “RoMA’s four perturbation positions” suggests all methods share the mask—then the 80% RA measures defense against attacks confined to regions the model was trained to defend. Other editable, functionality-preserving regions in PE files are not tested. That’s a real gap, but it’s a gap in generalization evidence, not evidence the method is hollow: within the tested attack surface, RoMA clearly beats FGSM-AT and PGD-based training by a wide margin.\n\nReporting is also thinner than it should be: no error bars, no repeated seeds, no validation split for hyperparameters, and the public artifact seems to be the model and sample images rather than training code. Those are fixable in revision. The C&W numbers show all adversarially trained methods are fine, which is credible.\n\nBottom line: worth a serious referee. I would ask for multi-seed runs, a clear statement of the evaluation attack mask, and at least one experiment with perturbations outside the four training regions before accepting the “over 80% robust accuracy” claim as a general statement. Not a desk reject; a conditional major revision.","headline":"A credible first pass at robust APT malware attribution with a large within-mask improvement, but the headline 80% robustness is only demonstrated under the four-region attack surface used in training.","tokens_in":14254,"tokens_out":2837,"would_cite":true,"duration_ms":28237,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoMA, a single-step adversarial training scheme, holds about 80% robust accuracy under PGD attacks on APT malware attribution while keeping clean accuracy near 91%.","keywords":["APT malware attribution","adversarial training","byte-level model","global perturbation","adversarial consistency regularization","PGD robustness","PE file","malware classification"],"falsifier":"Run PGD-50 with the same perturbation budget but permit byte changes in a fifth region outside the allowed four, such as the code section under a semantics-preserving transformation. If RoMA's robust accuracy drops from about 80% toward the 1-2% range seen for undefended MalConv, the robustness claim is limited to the paper's perturbation set rather than to advanced attacks in general.","tokens_in":13234,"feed_emoji":"🛡️","tokens_out":7274,"duration_ms":61994,"temperature":0.7,"pith_summary":"RoMA tries to establish that efficient single-step adversarial training can make raw-byte APT malware attribution robust to white-box attacks, closing the gap left by methods like MalConv, whose accuracy collapses from above 90% to below 2% under PGD. The method couples learned global perturbations in embedding space with two consistency losses, and the authors test it on a new 18-group APT dataset with realistic class imbalance. The paper's headline results are about 80% robust accuracy under PGD attacks, more than twice the best competing method, with higher clean accuracy and roughly half the training time of the second-best method. If these results hold, robust malware attribution no longer requires slow multi-step adversarial training or sacrificing non-adversarial performance.","feed_headline":"Malware attribution model keeps 80% accuracy under attack","feed_subtitle":"Single-step adversarial training more than doubles the strongest baseline's robust accuracy and trains twice as fast.","key_machinery":"The machinery is a pool of $K=50$ global perturbation vectors maintained in the word-embedding space. For each malware, a selection head chooses one GP via a contrastively trained logits layer; the GP is added to embeddings at the four perturbation positions and mapped back to nearest byte values, followed by a single FGSM gradient step that updates both the bytes and the selected GP with momentum. Around this, two auxiliary losses act on the projection space: adversarial contrastive loss pulls clean and adversarial projections of the same APT group together while pushing different groups apart, and adversarial distribution loss, a KL divergence, aligns the clean and adversarial prediction distributions.","core_discovery":"The central claim is that a byte-level APT malware attribution model trained with RoMA can withstand strong white-box PGD attacks while retaining clean-set accuracy and training faster than multi-step alternatives. The authors report 80.13% robust accuracy at PGD-50, falling to 78.79% at PGD-70, with attack success rate around 12-13%, compared with 36.85% RA and 58.86% ASR for the second-best PGD-4-AT; clean accuracy is 91.00% and training time is 2378 minutes, roughly half of PGD-4-AT. Against C&W attacks all adversarially trained models stay above 84% RA. The claimed mechanism is that single-step FGSM becomes sufficient when the perturbation is preceded by a learned global perturbation pattern and the representation is regularized by contrastive and distribution consistency losses.","pith_inferences":["Editorial inference: the four-region perturbation set defines the threat surface; expanding to other functionality-preserving regions, such as resources or relocation data, would likely change absolute numbers even if the relative ranking persists.","Editorial inference: the GP pool should be dataset-dependent; a model trained on AMG18 may need its GP pool re-initialized when applied to a different PE corpus, and that is a testable extension.","Editorial inference: ACLoss and ADLoss are not tied to attribution and could be tested on binary malware detection or binary authorship tasks with the same single-step training recipe."],"forward_implications":["Deployed attribution models can keep working under white-box PGD attacks: RoMA's RA stays near 80% at PGD-50 versus below 2% for standard MalConv.","Multi-step PGD training is not necessary for this task: RoMA trains in about 2378 minutes versus 4965 for PGD-4-AT while being more robust.","All three components matter: removing GP drops PGD-50 RA by 65.97 points, and removing all three drops RA by 69.80 points, so the robustness gain is attributed to the combination.","C&W optimization attacks are a weak test after adversarial training; all adversarially trained methods exceed 84% RA, so PGD remains the discriminating evaluation."],"supporting_citations":[{"why":"Supplies the MalConv-GCG architecture that RoMA uses as the trained attribution model.","marker":"[Raff et al., 2021]"},{"why":"Defines the PGD attack used for evaluation and the min-max formulation that motivates multi-step adversarial training.","marker":"[Madry et al., 2017]"},{"why":"Provides the PGD-based adversarial training for raw-byte malware classifiers that RoMA compares against as PGD-2-AT and PGD-4-AT.","marker":"[Lucas et al., 2023]"},{"why":"Introduces Slack-FGSM, the single-step adversarial training baseline that RoMA extends to four perturbation regions.","marker":"[Kreuk et al., 2018]"},{"why":"Introduces Append-FGSM, another single-step baseline subsumed in FGSM-AT.","marker":"[Kolosnjaji et al., 2018]"},{"why":"Supplies FGSM-RS, a single-step image-domain method adapted as a competitor in transfer learning.","marker":"[Wong et al., 2020]"},{"why":"Supplies NuAT, the nuclear-norm regularized single-step method adapted as a competitor.","marker":"[Sriramanan et al., 2021]"},{"why":"Provides the problem-space threat model that defines the attacker assumptions for functionality-preserving modifications.","marker":"[Pierazzi et al., 2020]"},{"why":"Supplies the C&W optimization attack used as a second robustness evaluation.","marker":"[Carlini and Wagner, 2017a]"},{"why":"Provides the APTMalware list that AMG18 extends with additional groups and samples.","marker":"[cyber research, 2019]"}],"fun_headline_variants":["New adversarial training keeps malware attribution 80% accurate under attack","Single-step adversarial training doubles malware attribution robustness","RoMA: malware attribution that stays accurate under adversarial attacks","New method makes malware attribution robust to PGD attacks with 80% accuracy","80% robust accuracy for malware attribution under white-box attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that an attacker's functionality-preserving byte modifications are confined to the four regions of a Windows executable that RoMA perturbs: the DOS header except its magic and PE pointer, a 1KB shift gap, slack space, and up to 100KB of appended padding.","fun_headline_variants_meta":{"raw":{"variants":["New adversarial training keeps malware attribution 80% accurate under attack","Single-step adversarial training doubles malware attribution robustness","RoMA: malware attribution that stays accurate under adversarial attacks","New method makes malware attribution robust to PGD attacks with 80% accuracy","80% robust accuracy for malware attribution under white-box attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000603,"raw_usage":{"total_tokens":2824,"prompt_tokens":968,"completion_tokens":1856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1773}},"tokens_in":584,"tokens_out":1856,"duration_ms":14055,"temperature":1.0,"reasoning_tokens":1773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:32:57.195695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PGD-50 with the same perturbation budget but permit byte changes in a fifth region outside the allowed four, such as the code section under a semantics-preserving transformation. If RoMA's robust accuracy drops from about 80% toward the 1-2% range seen for undefended MalConv, the robustness claim is limited to the paper's perturbation set rather than to advanced attacks in general.","supporting_citations":[{"cited_title":"Classifying sequences of extreme length with constant memory applied to malware detection","cited_arxiv_id":null,"evidence_quote":"Supplies the MalConv-GCG architecture that RoMA uses as the trained attribution model."},{"cited_title":"Reiter, and Mahmood Sharif","cited_arxiv_id":null,"evidence_quote":"Provides the PGD-based adversarial training for raw-byte malware classifiers that RoMA compares against as PGD-2-AT and PGD-4-AT."},{"cited_title":"Adversarial malware binaries: Evading deep learning for malware detection in executables","cited_arxiv_id":null,"evidence_quote":"Introduces Append-FGSM, another single-step baseline subsumed in FGSM-AT."},{"cited_title":"Fast is better than free: Revisiting adversarial training","cited_arxiv_id":null,"evidence_quote":"Supplies FGSM-RS, a single-step image-domain method adapted as a competitor in transfer learning."},{"cited_title":"Towards efficient and effective adversarial training","cited_arxiv_id":null,"evidence_quote":"Supplies NuAT, the nuclear-norm regularized single-step method adapted as a competitor."},{"cited_title":"Intriguing Properties of Adversarial ML Attacks in the Problem Space","cited_arxiv_id":null,"evidence_quote":"Provides the problem-space threat model that defines the attacker assumptions for functionality-preserving modifications."}],"review_version":1}