{"id":"64023f86-9f97-4bb1-a72d-6dfa7fdca579","arxiv_id":"2411.15878","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"ExAL uses EMPSO-generated perturbations for adversarial training and reports higher F1 for the adversarially trained CNN on MNIST and a blended malware dataset.","lead":"This paper proposes a method that uses a particle swarm optimizer variant, EMPSO, to generate adversarial perturbations, then retrains a CNN on those perturbed examples. It reports F1 scores on MNIST and a blended malware dataset, suggesting that the retrained model resists the generated attacks better than a model that was not retrained.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The robustness claim rests entirely on one fixed universal perturbation A*; no unseen or per-example attack is tested, so the secure-vs-manipulated gap may just be adaptation to a known shift rather than general adversarial robustness.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation uses a single perturbation A* generated once and applied identically to training and test data, with no test against unseen perturbations or other attack algorithms. My reading of the full text confirms that all reported evidence for robustness is contained in the F1 comparisons of CNNsecure and CNNmanipulated under this same A*, and that the paper does not include standard adversarial attack baselines, per-example attacks, transfer attacks, error bars, or multiple random seeds. The concern is therefore not merely a stylistic preference for more benchmarks; it is directly about whether the central claim—'significantly enhances model resilience to adversarial attacks'—is supported by the experimental design. The gap the paper highlights is expected whenever a model is trained and tested on the same fixed input transformation, so it cannot by itself distinguish genuine adversarial robustness from adaptation to a known distribution shift. A re-optimized perturbation or a standard attack on the secure model would settle this. Because the reader's verdict is already CONDITIONAL and this concern is the same one that motivated that condition, no verdict change is needed; the conditional should remain until the proposed robustness checks are run.","tokens_in":11752,"tokens_out":4014,"duration_ms":41393,"concrete_test":"Run a second EMPSO/ExAL optimization against CNNsecure on the training set to obtain a new perturbation B*, then evaluate CNNsecure on Xtest + B*. If F1 drops to or below CNNmanipulated's level on Xtest + A*, the original gap is specific to A* rather than evidence of general robustness. Complement this with a standard per-example attack (e.g., PGD or FGSM) on both CNNsecure and CNNmanipulated, and compare against a model trained on Xtrain plus a random constant shift of the same magnitude as A*, to isolate whether ExAL's exploration mechanism adds anything beyond ordinary adaptation to a known shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ExAL 'significantly enhances model resilience to adversarial attacks' is supported only by Tables 1 and 2, which compare CNNsecure = CNN(Xtrain + A*, Xtest + A*) with CNNmanipulated = CNN(Xtrain, Xtest + A*) as defined in Box 2 and Section 3.2. Here A* is a single universal perturbation vector produced once by EMPSO against the original clean-trained model, then added identically to every training and test example. Training on Xtrain + A* and testing on Xtest + A* places the secure model on the same shifted distribution it was trained on, so the reported F1 gap relative to the clean-trained manipulated model is essentially a covariate-shift adaptation effect. No adversarial examples are generated against CNNsecure at test time, no per-example attack such as FGSM or PGD is considered, and no independently re-optimized perturbation is evaluated. Thus the experiments do not establish robustness to adversarial attacks beyond this one fixed A*, leaving the abstract's general 'resilience' claim underdetermined. The absence of error bars and multiple seeds further weakens the quantitative comparison, although the consistency of the F1 ordering across all twelve rows partially mitigates that concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ExAL, an adversarial learning method that uses the Exponentially Weighted Momentum Particle Swarm Optimizer (EMPSO) to optimize a single universal perturbation A* against a clean-trained CNN, then retrains the CNN on the perturbed training set. The authors evaluate three models—CNNoriginal, CNNmanipulated (clean-trained, tested on perturbed test data), and CNNsecure (retrained and tested on the same perturbed data)—on six binary label pairs from MNIST and six from the Blended Malware dataset, reporting F1 scores. The central empirical claim is that the secure model consistently outperforms the manipulated model, which the paper interprets as evidence that ExAL improves adversarial robustness.","tokens_in":12081,"tokens_out":5323,"duration_ms":50755,"significance":"The idea of using PSO-based exploration to generate adversarial perturbations for adversarial training is potentially interesting, and the paper gives a fairly detailed algorithmic description. However, as it stands, the experiments support only a narrow covariate-shift adaptation result: training and testing on the same fixed perturbation improves F1 relative to a clean model tested on that same perturbation. The broader claim that ExAL 'significantly enhances model resilience to adversarial attacks' is not yet supported because no unseen perturbations, no per-example attacks, and no standard adversarial training baselines are evaluated. If the authors substantially extend the experimental evaluation and align the optimization objective with the stated game-theoretic formulation, the contribution could become a useful incremental result in adversarial learning.","major_comments":[{"comment":"The evaluation uses a single fixed perturbation A* optimized once against the original model. CNNsecure is trained and evaluated on X+A*, while CNNmanipulated is evaluated only on the same Xtest+A*. This setup measures adaptation to a known distribution shift, not robustness to adversarial attacks in any general sense: no adversarial examples are generated against CNNsecure at test time, no per-example attack such as FGSM or PGD is considered, and no independently re-optimized perturbation is tested. The claim in the abstract that ExAL 'significantly enhances model resilience to adversarial attacks' is therefore underdetermined. Please add evaluations under unseen perturbations (for example, a re-optimized A*, FGSM, and PGD) and restrict the conclusions to the specific perturbation used in training until such evidence is provided.","section":"§4, Boxes 2 and 4, Tables 1–2"},{"comment":"There are no comparison baselines such as standard adversarial training (FGSM/PGD-based), no repeated runs, and no error bars or significance tests. The statement in Section 6 that ExAL 'outperforms baseline models' is unsupported because no baseline adversarial training method is included in the experiments. The consistent ordering across all twelve rows is suggestive, but without variance estimates and baseline comparisons, the size and reliability of the reported improvements cannot be assessed.","section":"§4, Tables 1–2"},{"comment":"There is an inconsistency between the mathematical formulation and the implemented fitness objective. Equation (7) states that A* = arg max_A L_adversary, i.e., the adversary maximizes the learner's loss, but Algorithm 4 computes the payoff as π = 1 + e − c, where c = ||a||_2. This means the optimized objective is e − ||a||_2, an unweighted combination of classification error and perturbation norm, not the learner's loss. The trade-off between these two terms is arbitrary and not derived from the game formulation. Please align the objective with Eq. (6)–(7) or explain and justify the penalty term.","section":"§3.2 vs. Algorithm 4"},{"comment":"The pseudocode for the momentum update is internally inconsistent. Line 14 updates v_i using βm_i + (1−β)v_i plus cognitive and social terms, and line 16 then updates m_i as βm_i + (1−β)v_i. If the updated v_i is used on the right-hand side of line 16, the momentum update no longer matches Eq. (2); if the old v_i is intended, the pseudocode should use a temporary variable to avoid ambiguity. This is load-bearing because the paper's contribution is precisely the EMPSO update, and as written the algorithm is not reproducible.","section":"Algorithm 6, lines 14–16"}],"minor_comments":[{"comment":"There is a typo in line 3: 'cognititive' should be 'cognitive'.","section":"Algorithm 2"},{"comment":"The model names are inconsistent: Box 3 uses 'CN Noriginal', while the tables and text use 'CNNorg', 'CNNmanip', and 'CNNsec'. Please define and use one consistent notation throughout.","section":"§4.1, Box 3 and Tables 1–2"},{"comment":"The line 'r ← M.evaluate(Xadv, Ytrain)' is ambiguous: model evaluation typically returns loss and possibly several metrics, not directly a recall value. Please specify how recall is computed and which metric is used for the binary classification tasks.","section":"Algorithm 4, line 7"},{"comment":"No hyperparameter values are reported for np, β, c1, c2, Tmax, the perturbation bounds beyond the MNIST value of ±0.1, or the CNN architecture and training hyperparameters. Without these details, the experiments cannot be reproduced.","section":"§4.1"},{"comment":"The related-work discussion is broad but omits standard adversarial training baselines (e.g., Madry et al. and Goodfellow et al.'s FGSM training) that are directly relevant to the claimed improvement; citing and comparing against these would strengthen the positioning of ExAL.","section":"§2.1 and §5"}],"recommendation":"major_revision","confidential_remarks":"The central issue is not circularity in the derivation-to-input sense but the under-specified evaluation: the reported secure-vs-manipulated gap may reflect adaptation to a known shift rather than general adversarial robustness. The paper would need a substantially extended experimental section with unseen attacks and standard baselines to support the abstract's claims. The authors should also double-check the momentum update in Algorithm 6 and the mismatch between Eq. (7) and the payoff in Algorithm 4."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ExAL is a small but honest step within an existing research line: the authors take the two-player adversarial game from Chivukula and Liu (2019), swap simulated annealing for EMPSO, and show that training a CNN on the resulting universal perturbation improves F1 on that same perturbation. The pseudo-code is complete, the writing is clear, and the F1 ordering (CNNsecure above CNNmanipulated) is consistent across all twelve label pairs. That is real, though modest, evidence of a covariate-shift adaptation effect. What the paper does not do is support the abstract's claim of 'significantly enhances model resilience to adversarial attacks.' The evaluation uses a single fixed A* that is optimized once against the clean model and then applied identically to both training and test sets. CNNsecure is simply a model trained and evaluated on the same shifted distribution. No per-example attack, no PGD or FGSM baseline, no re-optimized perturbation, and no adversarial example crafted against CNNsecure at test time. In other words, the robustness claim is untested. The absence of error bars or multiple seeds also makes the F1 gaps impossible to interpret for the smaller differences (e.g., 0.8628 vs 0.8838). I would have liked to see a direct comparison to the original SA-based formulation, and at least a discussion of hyperparameters (beta, c1, c2, swarm size, bounds) so the experiments could be replicated. The exploration-related motivation is asserted, not demonstrated; there is no measure of exploration behavior. All of that said, the paper does not cheat: it clearly describes what was done, the prior work is cited fairly, and the negative results are not hidden. The central idea is a legitimate extension, but the evidence for the central claim is weak. This is a paper that a workshop might accept after moderate revision, not a strong journal paper. For a serious journal or conference, I would send it to reviewers only with the explicit instruction that the authors need to add standard adversarial training baselines and test against unseen attacks; otherwise the main claim remains unsupported.","headline":"A cleanly described but narrowly evaluated variant of an existing adversarial game; the robustness claim rests on one fixed universal perturbation, so it is over-sold.","tokens_in":12580,"tokens_out":2291,"would_cite":false,"duration_ms":20876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ExAL claims that training a CNN on EMPSO-generated adversarial perturbations raises F1 scores on all twelve binary tasks.","keywords":["adversarial learning","particle swarm optimization","EMPSO","adversarial robustness","MNIST","malware detection","convolutional neural network","exploration"],"falsifier":"Run a separate attack (for example, a fast gradient sign or projected gradient descent step) on CNNsecure and compare its F1 against an ordinary CNN; if the ExAL-trained model does not hold up against an attack it was not trained on, the paper's claim of general resilience fails. Equally, re-optimizing A* per test batch rather than using the single fixed vector would test whether the reported robustness is specific to A*.","tokens_in":1573,"feed_emoji":"🛡️","tokens_out":2141,"duration_ms":62176,"temperature":0.7,"pith_summary":"ExAL is an adversarial training method that replaces simulated annealing with a particle-swarm optimizer, EMPSO, to produce a single adversarial perturbation vector. The paper's central claim is that training a CNN on this perturbed data makes the model more resilient: in every one of the twelve binary classification tasks reported, the ExAL-trained model achieves a higher F1 score than an identical model evaluated on the same perturbed test data. The perturbations are bounded to [-0.1, 0.1] per pixel, so the method aims to keep them visually subtle while maximally confusing the classifier. If the claim holds, ExAL offers a way to harden classifiers against evasion attacks using a comparatively simple optimization step.","feed_headline":"Particle-swarm adversarial training lifts CNN F1 on every test pair","feed_subtitle":"ExAL's secure CNN outscored models evaluated on the same perturbed data across six MNIST and six malware label pairs.","key_machinery":"The central object is EMPSO, an Exponentially Weighted Momentum Particle Swarm Optimizer. Its velocity update rule combines an exponentially weighted momentum term m_i <- beta m_i + (1-beta) v_i with cognitive and social attraction terms, and the global best velocity v*_gbest is returned as the adversarial perturbation A*. The fitness of each candidate perturbation is the negative adversary payoff, payoff = 1 + (1 - recall) - ||a||_2, so the swarm seeks perturbations that raise classification error while staying small. EMPSO does the work of converting model loss into a concrete perturbation vector used in both attack and defense.","core_discovery":"The paper establishes that a single perturbation vector A*, found by EMPSO against the original model's weights, can be reused twice: once to augment training data and once to evaluate robustness. With that vector, CNNsecure = CNN(Xtrain + A*, Xtest + A*) outperforms CNNmanipulated = CNN(Xtrain, Xtest + A*) on all six MNIST label pairs (Table 1) and all six Blended Malware pairs (Table 2). The hypothesis boxed in the paper—that the adversary hones perturbations that erode the learner's accuracy—is reported as satisfied in every case. The mechanism is the EMPSO velocity update with exponential momentum, guided by a fitness function equal to minus the adversary payoff, where payoff is 1 plus classification error minus the L2 norm of the perturbation.","pith_inferences":["A natural extension would compare ExAL against standard adversarial training methods that use per-sample gradient-based perturbations, since the paper does not include such a baseline.","Because the same A* is used for both training augmentation and test evaluation, the reported numbers may partly reflect memorization of one perturbation direction; testing with fresh EMPSO runs per batch would clarify.","The swarm fitness deliberately minimizes the L2 norm of the perturbation, so ExAL should naturally favor imperceptible attacks; this makes it a candidate for low-budget adversarial training in settings where visible corruption is unacceptable.","Applying ExAL to larger, more diverse datasets would reveal whether the single-vector strategy scales beyond the 1,000-sample binary tasks reported here."],"forward_implications":["A CNN trained on ExAL-generated perturbed data (CNNsecure) is reported to achieve F1 scores at or near those of the original model even when the same perturbation is applied at test time.","ExAL-generated perturbations are bounded per pixel to [-0.1, 0.1], so the defense is explored in a regime where perturbations remain visually subtle.","The pattern holds across two different data modalities—handwritten digit images and blended malware images—suggesting the method is not tied to one domain.","Increasing the perturbation scale from 0.5 to 5 makes the attack stronger against the manipulated model, but the secure model still reports near-perfect F1 on most malware pairs."],"supporting_citations":[{"why":"Supplies the two-player adversarial game setup and the CNNoriginal, CNNmanipulated, and CNNsecure model definitions that ExAL builds on while replacing simulated annealing with EMPSO.","marker":"Chivukula and Liu (2019)"},{"why":"Source of the EMPSO optimizer whose velocity and momentum update rules ExAL uses to generate adversarial perturbations.","marker":"Mohapatra et al. (2022)"},{"why":"Motivates adversarial examples and the GAN-based view of adversarial learning that frames the paper's approach.","marker":"Goodfellow et al. (2014)"},{"why":"Defines the base particle swarm optimization algorithm that EMPSO extends with momentum.","marker":"Kennedy and Eberhart (1995)"},{"why":"Defines simulated annealing, the prior optimizer used in the adversarial learning game that ExAL replaces.","marker":"Kirkpatrick et al. (1983)"},{"why":"Provides the MNIST Handwritten Digits dataset used in Experiment 1.","marker":"LeCun et al. (1998)"},{"why":"Provides the Blended Malware Image Dataset used in Experiment 2.","marker":"Pendharkar (2021)"}],"fun_headline_variants":["One adversarial vector trains and tests ExAL's robust CNN","EMPSO finds a single perturbation that boosts defense on all pairs","ExAL reuses one attack vector to upgrade CNN robustness across datasets","Game-theoretic ExAL: same perturbation augments training and evaluation","Exponential momentum optimizer yields CNN win in every label pair test"],"cache_read_input_tokens":14720,"weakest_assumption_plain":"The evaluation assumes that a single perturbation vector A*, optimized once against the original training model, transfers unchanged to the test set and fully represents what an adversary can do; if a different attack algorithm or a fresh per-sample perturbation would break the secure model, the general robustness claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["One adversarial vector trains and tests ExAL's robust CNN","EMPSO finds a single perturbation that boosts defense on all pairs","ExAL reuses one attack vector to upgrade CNN robustness across datasets","Game-theoretic ExAL: same perturbation augments training and evaluation","Exponential momentum optimizer yields CNN win in every label pair test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1392,"prompt_tokens":895,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":511,"tokens_out":497,"duration_ms":5140,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:46:15.074575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a separate attack (for example, a fast gradient sign or projected gradient descent step) on CNNsecure and compare its F1 against an ordinary CNN; if the ExAL-trained model does not hold up against an attack it was not trained on, the paper's claim of general resilience fails. Equally, re-optimizing A* per test batch rather than using the single fixed vector would test whether the reported robustness is specific to A*.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the two-player adversarial game setup and the CNNoriginal, CNNmanipulated, and CNNsecure model definitions that ExAL builds on while replacing simulated annealing with EMPSO."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the EMPSO optimizer whose velocity and momentum update rules ExAL uses to generate adversarial perturbations."},{"cited_title":"J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"Motivates adversarial examples and the GAN-based view of adversarial learning that frames the paper's approach."},{"cited_title":"and Eberhart, R","cited_arxiv_id":null,"evidence_quote":"Defines the base particle swarm optimization algorithm that EMPSO extends with momentum."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines simulated annealing, the prior optimizer used in the adversarial learning game that ExAL replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MNIST Handwritten Digits dataset used in Experiment 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Blended Malware Image Dataset used in Experiment 2."}],"review_version":1}