{"id":"8295bc8e-b07e-466f-a366-247532933814","arxiv_id":"2411.09296","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Adversarial training reduces loss sharpness and improves cross-Monte-Carlo generalization for Higgs-jet classifiers, with projected gradient descent giving the largest gains.","lead":"This paper tests whether adversarial training, a technique that makes neural networks robust to small perturbations, helps particle-physics classifiers stop over-relying on Monte Carlo simulation details. It shows the technique improves performance when a classifier trained on one simulated dataset is evaluated on another, suggesting the same may hold on real data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed 24h training budget is a confound: default and adversarial models receive different numbers of gradient updates, so Table V cross-simulator gains may be an epoch/overfitting effect rather than evidence that white-box attacks improve generalization.","rationale":"I selected the training-budget confound as the most load-bearing concern because it directly threatens whether the measured improvement is caused by adversarial perturbation at all. The proxy-to-real-data issue identified by the reader is real but is an external-validity question; the time-limit issue is an internal-validity question about the experiment's headline table. The authors deserve credit for public code/data, seed averaging, and tuning only on training data, and their sharpness analysis is internally consistent. However, the fixed 24 h limit, introduced to make the comparison 'fair' in wall-clock terms, means the methods are not compared at equal training effort. Since the paper's conclusion is explicitly that adversarial strategies 'resulted in a significant increase in generalization performance,' this confound must be controlled before the causal claim is accepted. The proposed equal-update-count reproduction is cheap, uses the released code, and would settle the concern. Until then, the appropriate verdict remains conditional: the cross-simulator improvement is promising but not yet causally attributed to adversarial training. I partially agree with the reader's weakest assumption: the real-data proxy is a valid long-term concern, but the more immediate threat to the central claim is whether the controlled comparison actually isolates the adversarial mechanism.","tokens_in":15930,"tokens_out":5234,"duration_ms":51052,"concrete_test":"Reproduce Table V with the number of optimizer updates matched across all methods: train default, FGSM, PGD, SAM, and SSAM-D for the same number of epochs on Pythia, and also include an early-stopped default baseline at the epoch where PGD stops. Compute cross-Pythia/Herwig AUC and ΔS for each. If PGD no longer significantly outperforms default, or if the early-stopped default matches the adversarial gains, the headline result is an artifact of unequal training time rather than adversarial sharpness reduction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim, Table V showing all four adversarial methods improve cross-simulator performance, depends on the comparison being a fair test of the training algorithm. The protocol in Sec. V imposes a fixed wall-clock limit of 24 h, and hyperparameters are tuned on the training set only (App. C2), but the per-batch cost of FGSM, PGD (5 steps), SAM, and SSAM-D differs substantially from that of plain SGD/AdamW. Under a fixed time budget, each method therefore receives a different number of optimizer updates. Default training is the cheapest per batch, so it likely gets the most epochs and has the most opportunity to overfit to Pythia-specific simulation artifacts; PGD is the most expensive per batch and may be implicitly regularized by receiving far fewer updates. The sharpness measurements in Table IV show PGD flattens feature-space loss, and the paper elsewhere attributes part of the cross-evaluation asymmetry to mass correlation (App. D2), but neither observation rules out the time-budget confound. If one reran the comparison with the number of gradient updates held fixed, the gap between default and adversarial methods could vanish. The paper's own 'fair practical comparison' wording makes the budget a deliberate treatment variable, which is exactly the kind of hidden intervention that could produce the reported improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether four white-box adversarial training methods (FGSM, PGD, SAM, and SSAM-D) reduce the sharpness of the loss minimum and improve cross-simulator generalization of a dense jet tagger. The classifiers are trained on the Pythia-based RS3L simulation and evaluated on the Herwig-based RS3L4 simulation, and vice versa, with AUC and background rejection as metrics. The paper proposes gradient-ascent path tracing and a reduced Hessian eigenvalue analysis as sharpness diagnostics, and reports that all four adversarial methods improve cross-evaluation performance, with PGD performing best, at increased computational cost.","tokens_in":16164,"tokens_out":6130,"duration_ms":103539,"significance":"If the central comparison is valid, this is a practically useful result for HEP classification: it suggests that training with norm-constrained perturbations of inputs or weights can reduce sensitivity to Monte Carlo showering details and improve OOD transfer between simulators. The paper has genuine strengths: the data and code are public, adversarial hyperparameters are tuned on the training distribution only, results are averaged over five seeds, and the sharpness diagnostics are computed independently of the generalization scores. The authors also explicitly flag that validation on real collision data remains future work. However, the fixed 24-hour wall-clock protocol and the noise-level of some raw-score gains need to be addressed before the headline claim can be accepted.","major_comments":[{"comment":"The fixed wall-clock budget is a treatment confound. The per-batch cost differs substantially across methods: default AdamW requires one forward-backward pass, FGSM two backward passes, PGD five additional forward-backward iterations, SAM two full backward passes, and SSAM-D one full plus one partial backward pass. Under a fixed 24-hour budget each method therefore receives a different number of optimizer updates, and the paper does not report epoch counts or total update counts per method. The Table V cross-evaluation gains could be due to a form of implicit early stopping (fewer gradient updates for the more expensive methods) rather than to flat minima induced by adversarial training. Please report the number of epochs/update steps per method and add at least one matched-update control (for example, the same number of optimizer steps for default and PGD), or otherwise show that the cross-evaluation improvement survives when update counts are held fixed.","section":"Sec. V, fixed 24 h limit; Sec. VI C, Table V"},{"comment":"The conclusion that \"all implemented adversarial strategies resulted in a significant increase in generalization performance\" is not supported by the raw scores for FGSM in the Pythia-to-Herwig direction: the raw AUC is 0.957 +/- 0.004 versus 0.953 +/- 0.004 for default, and the rejection is 11.5 +/- 0.1 versus 11.3 +/- 0.2. These differences are within the reported seed standard deviations. The normalized Delta-S value of 0.21 +/- 0.01 in Table V appears much more significant than the raw comparison; the error propagation for Delta-S should be explained, including whether the five seeds were paired between methods. Please provide per-seed paired differences or a significance test on the raw scores for each method and direction.","section":"Sec. VI C, Tables V and IX"},{"comment":"The Hessian analysis is performed on a reduced subspace: only the first five jet constituents are considered in feature space, and only the final classification layer is considered in weight space. The text draws conclusions such as \"FGSM and PGD significantly reduce feature-space sharpness\" and \"SAM and SSAM-D reduce weight-space sharpness.\" Those statements are strictly supported only for the reduced subspaces, not for the full loss landscape. The claim in Appendix A that the first five constituents are a \"reasonable\" approximation is not quantified. Please either soften the sharpness claims to refer to the reduced subspaces or provide evidence (for example, sensitivity to the number of constituents and layers) that the conclusions are stable under the truncation.","section":"Sec. IV C, Appendix A, Table IV"},{"comment":"The empirical evaluation rests on the assumption that discrepancies between Pythia and Herwig for the same process proxy the simulation-to-data shift that matters in ATLAS analyses. The authors acknowledge this point and correctly place real-data validation in future work, so the assumption is not hidden. Nevertheless, the abstract and conclusion should be worded so that cross-simulator improvement is not read as demonstrated improvement on real collision data; the current phrasing \"generalize well to real data\" in the abstract is stronger than the evidence presented.","section":"Sec. II, Eq. (3); Sec. VII"}],"minor_comments":[{"comment":"In the paragraph after Fig. 3, \"PDG\" appears twice where the intended abbreviation is \"PGD\" (projected gradient descent); please correct the typo.","section":"Sec. VI B"},{"comment":"The text says \"Sharpness-aware methods, especially SSAMD, results in a slightly improved performance average\" in Appendix D1; the subject-verb agreement should be fixed, and the sentence should specify whether the improvement is relative to default and over which efficiency range.","section":"Sec. VI C and Appendix D1"},{"comment":"The power iteration is called the \"von Mises iteration algorithm\" and Ref. [41] is cited; this is correct, but the Rayleigh quotient expression in Eq. (A4) is missing a normalizing denominator and should read lambda_k = (b_k^T H_L b_k)/(b_k^T b_k) for clarity.","section":"Appendix A, Eq. (A3)"},{"comment":"The notation R_OOD(f) is introduced as the maximum risk over all environments, then approximated by the maximum over training environments; this is an unusual use of \"OOD\" and deserves a sentence clarifying that the maximum over training environments is only a proxy for the unknown OOD maximum.","section":"Sec. II, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"I largely agree with the conditional assessment: the central claim is plausible and the experimental protocol is careful, but the fixed 24-hour budget is a genuine confound that must be addressed with matched-update controls. The FGSM raw-score significance issue is also worth requiring the authors to fix. If the authors can supply equal-update comparisons and a clearer significance analysis, the paper would be a solid contribution to the HEP-ML literature; otherwise the headline generalization claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a careful, useful engineering study. The headline is that adversarial training—PGD in particular—improves cross-simulator jet tagging performance, and the sharpness diagnostics give a plausible mechanism. The empirical pattern is clean: feature-space attacks flatten feature-space Hessian eigenvalues, weight-space attacks flatten weight-space ones, and this holds across all cross-evaluation settings. That selectivity is the paper's best evidence that the perturbation space, not an incidental training difference, is doing the work.\n\nWhat's actually new: Ref [19] used FGSM only; this adds PGD, SAM, and SSAM-D, and replaces the adversarial test set with two independent simulators (Pythia/Herwig) as a validation proxy. The public code/data, five seeds, and training-set-only hyperparameter search make the protocol easy to trust.\n\nNow the soft spots. The fixed 24 h budget is a real confound. Default training is the cheapest per update, so it almost certainly gets more epochs and more chance to overfit to Pythia; PGD is the most expensive, so it is implicitly regularized by receiving fewer updates. The selective sharpness pattern makes it unlikely that the entire gap is an epoch effect, but the paper never reports epochs or update counts, so the claim \"adversarial training improves generalization\" is not cleanly separated from \"fewer updates improves generalization.\" A controlled run with matched update counts, or early stopping for the baseline, would settle it.\n\nThe second limitation is external validity. The cross-MC proxy is reasonable, but it is never checked against real ATLAS data. The authors acknowledge this in the conclusion; still, the title says \"enhancing generalization in high energy physics\" when the evidence is really cross-simulator. The Hessian approximations (first five constituents, last layer) are coarse but justified in the appendix.\n\nOverall, this is a solid comparative study, more useful to practitioners than to theorists. I'd send it to peer review—it deserves referee time—but with a request to address the update-count confound and to soften the generalization language. I'd cite it for the PGD-vs-SAM comparison and the sharpness analysis.","headline":"Solid engineering study: PGD adversarial training improves cross-simulator jet tagging with a clean sharpness signature, but the fixed 24h budget and unvalidated cross-MC proxy keep the headline claim from being fully settled.","tokens_in":16764,"tokens_out":4457,"would_cite":true,"duration_ms":40277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"White-box adversarial training improves out-of-distribution generalization for Higgs jet classifiers by flattening sharp loss minima.","keywords":["adversarial training","jet tagging","out-of-distribution generalization","loss landscape sharpness","Monte Carlo simulation","Higgs boson","projected gradient descent","sharpness-aware minimization"],"falsifier":"Train the default model and the projected-gradient-descent adversarial model on the resimulation dataset and evaluate both on a real collision-data control sample enriched in the same Higgs decay and background processes at a fixed signal efficiency. If the adversarial model's advantage over default on the other simulator does not appear on the real-data control sample, the central proxy assumption is falsified.","tokens_in":15672,"feed_emoji":"⚛️","tokens_out":9784,"duration_ms":100600,"temperature":0.7,"pith_summary":"Supervised classifiers trained on Monte Carlo simulations can latch onto artifacts of the simulation rather than the underlying physics, so they lose performance when applied to a different simulator or to real data. This paper argues that this poor out-of-distribution behavior is tied to the sharpness of the loss minimum the network settles into, and that white-box adversarial training, which perturbs inputs or weights using the model's own gradients, reduces that sharpness. It tests four adversarial strategies on Higgs-to-bottom-quark jet tagging, using cross-evaluation between two different Monte Carlo simulators as the generalization test. All four strategies improve cross-simulator performance, and the projected gradient descent attack gives the largest gains. The authors treat the simulator-to-simulator gap as a proxy for the simulation-to-data gap, so the ultimate claim about real data remains to be verified.","feed_headline":"White-box attacks improve how jet taggers transfer between simulators","feed_subtitle":"Perturbing inputs during training flattens the loss landscape, reducing overfitting to Monte Carlo artifacts.","key_machinery":"The load-bearing mechanism is the sharpness of the local minimum in the training loss, defined by comparing loss increases under norm-bounded perturbations, with sharper minima expected to generalize worse. To reduce sharpness, the paper uses four white-box adversarial training methods: fast gradient sign method and projected gradient descent, which maximize the loss over perturbations of the input features, and sharpness-aware minimization and its dynamic sparse variant, which maximize the loss over perturbations of the network weights. To measure sharpness, the paper introduces gradient-ascent path tracing, which plots the loss increase as a projected gradient ascent walks away from the minimum, and reduced Hessian eigenvalue analysis, which estimates the largest Hessian eigenvalue via power iteration as a sharpness proxy. The Hessian is computed on a reduced subspace: the first five jet constituents for feature space and the final classification layer for weight space.","core_discovery":"The paper's central claim is that training a jet classifier with norm-bounded adversarial perturbations improves its out-of-distribution generalization because it drives the optimizer toward flatter, wider local minima of the loss landscape. The authors establish this in two steps: they show, with gradient-ascent curves and largest Hessian eigenvalues, that each adversarial strategy measurably flattens the relevant minimum, and they show, by cross-evaluating models trained on one Monte Carlo simulator and tested on another, that every strategy improves the generalization metrics relative to default training. Feature-space attacks flatten feature-space sharpness, weight-space attacks flatten weight-space sharpness, and the two kinds of sharpness are not correlated; projected gradient descent is the strongest method overall. Because the evaluation never uses real collision data, the paper's conclusion is conditional: the observed generalization gains support the sharpness hypothesis only insofar as cross-simulator discrepancy represents the real simulation-to-data discrepancy.","pith_inferences":["If the simulator-to-simulator gap is representative of the simulation-to-data gap, the same training recipes should improve performance on real collision data, which is a direct test of the paper's central assumption.","Because feature-space and weight-space sharpness appear independent, combining an input-perturbation method with a weight-perturbation method may yield larger gains than either family alone.","The mechanism is not specific to Higgs jet tagging, so similar cross-simulator gaps in other collider-classification tasks may be reduced by the same recipes after re-tuning the perturbation radii."],"forward_implications":["Training with projected gradient descent improves cross-simulator AUC and background rejection relative to default training in both transfer directions, and it outperforms fast gradient sign training in every scenario tested.","Feature-space adversarial training systematically reduces the largest Hessian eigenvalue in feature space, while weight-space training reduces it in weight space; the two sharpness measures move independently.","All four strategies increase computational cost, and the comparison is run under a fixed 24-hour training budget per model.","The ranking of the strategies is stable across the signal-efficiency spectrum, not only at the single operating point used for the headline numbers."],"supporting_citations":[{"why":"Supplies the resimulation-based jet dataset whose two simulator versions define the environments used for the generalization test.","marker":"[42]"},{"why":"The earlier FGSM-based jet-tagging robustness study that this paper extends with additional attacks and sharpness quantification.","marker":"[19]"},{"why":"Provides the out-of-distribution risk formalism that motivates treating simulator discrepancy as a generalization proxy.","marker":"[20]"},{"why":"The source of the sharp-minima generalization-gap argument that the paper's sharpness reduction builds on.","marker":"[24]"},{"why":"Introduces the fast gradient sign method, one of the four adversarial training strategies tested.","marker":"[33]"},{"why":"Introduces projected gradient descent, the attack that gives the largest generalization improvements in this paper.","marker":"[36]"},{"why":"Introduces sharpness-aware minimization, the weight-space attack based on perturbing weights.","marker":"[37]"},{"why":"Introduces the dynamic sparse SAM variant used as the fourth training strategy.","marker":"[40]"},{"why":"Defines the nominal parton-shower simulator used for the primary training environment.","marker":"[44]"},{"why":"Defines the second simulator used as the cross-evaluation environment; its different showering produces the distribution shift the paper exploits.","marker":"[46]"}],"fun_headline_variants":["Adversarial attacks flatten loss landscape, boost jet tagger transfer","White-box attacks reduce Monte Carlo overfitting in jet tagging","Flatter minima via adversarial training improve simulation transfer","Perturbing inputs during training strengthens jet tagger generalization","Adversarial training in HEP: less sharp minima, better transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that differences between two Monte Carlo simulations of the same physics process stand in for the difference between simulation and real detector data; if that proxy is unrepresentative, the reported cross-simulator gains may not transfer to real collision data.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial attacks flatten loss landscape, boost jet tagger transfer","White-box attacks reduce Monte Carlo overfitting in jet tagging","Flatter minima via adversarial training improve simulation transfer","Perturbing inputs during training strengthens jet tagger generalization","Adversarial training in HEP: less sharp minima, better transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2203,"prompt_tokens":875,"completion_tokens":1328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1257}},"tokens_in":491,"tokens_out":1328,"duration_ms":9914,"temperature":1.0,"reasoning_tokens":1257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:48:21.922838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the default model and the projected-gradient-descent adversarial model on the resimulation dataset and evaluate both on a real collision-data control sample enriched in the same Higgs decay and background processes at a fixed signal efficiency. If the adversarial model's advantage over default on the other simulator does not appear on the real-data control sample, the central proxy assumption is falsified.","supporting_citations":[{"cited_title":"Zhang, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the resimulation-based jet dataset whose two simulator versions define the environments used for the generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier FGSM-based jet-tagging robustness study that this paper extends with additional attacks and sharpness quantification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The source of the sharp-minima generalization-gap argument that the paper's sharpness reduction builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces sharpness-aware minimization, the weight-space attack based on perturbing weights."}],"review_version":1}