{"id":"7d15cd71-4078-402b-a9eb-1317e1dabd6d","arxiv_id":"2607.16348","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"BGAN-augmented TabTransformer improves Macro-F1 on CICIDS2017 from 82.96% to 86.50%, but the reported adversarial robustness gains are self-referential because the test uses the same BGAN samples used for training.","lead":"A machine-learning intrusion detection system gains better detection of rare attacks on a standard benchmark after adding GAN-generated synthetic attack samples to its training data. The paper's claim that this also makes the system robust to adversarial attacks is undermined by a test design that uses the same generator for both augmentation and evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PDR evaluation is circular: adversarial test samples are drawn from the same BGAN used to augment training, so negative PDR likely measures in-distribution memorization rather than adversarial robustness.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: using the same BGAN for augmentation and adversarial testing makes the robustness evaluation circular. I confirm this from the manuscript: Section 3.2 describes the dual-purpose BGAN, Section 3.5 defines PDR over BGAN-generated adversarial samples, and Table 6 reports the results. The paper's own future-work statement in Section 6 concedes that an independent generator or standard attack is needed, which is strong evidence that the current evaluation is not a valid adversarial test. Negative PDR values are a red flag because adversarial samples should be harder than clean test samples, not easier; their presence indicates the generated samples are in-distribution for the augmented classifier. The clean-data Macro-F1 improvement (82.96% to 86.50%) is plausible and is not the focus of this objection, but the paper's central contribution is explicitly adversarial robustness. The cited external evidence about Transformer robustness ([9], [8]) cannot rescue the primary evaluation because those studies do not test this BGAN-augmentation protocol or this circular PDR design. Thus the central claim is unsupported as presented. A revised evaluation with independent attacks could change this, but as written the paper should be rejected. I agree with the reader's verdict and reasoning; no stronger concern is needed.","tokens_in":11718,"tokens_out":3777,"duration_ms":45783,"concrete_test":"Recompute Table 6 using a training-independent attack instead of the augmentation BGAN. Specifically, generate PGD adversarial examples on the held-out test set with a fixed L-infinity budget (e.g., epsilon = 0.1 in the standardized feature space) for each trained classifier, and recompute PDR via Eq. (8). If Full-BGAN models still show negative or near-zero PDR under this independent attack, the robustness conclusion is supported; if they show large positive PDR, the current negative PDR is an artifact of the attack generator being included in the training set. A secondary confirmation: train a second BGAN on the original training set, never use it for augmentation, and test all models against it to see whether non-augmented models still collapse and augmented models still improve.","verdict_should_be":"REJECT","load_bearing_attack":"The central robustness claim rests on the PDR results in Table 6, computed via Eq. (8) as F1_clean minus F1_adversarial over clean. Section 3.5 defines the adversarial samples as BGAN-generated samples of the same class, and Section 3.2 states that the same trained BGAN is used both to augment the training set and to generate these adversarial samples. For every Full-BGAN model, the attack distribution is therefore a subset of the training distribution: the classifier was trained to label outputs of that exact generator, so high F1 on those samples is expected regardless of robustness to genuine adversarial perturbation. The negative PDR values (e.g., TabTransformer Full BGAN -24.75% on Botnet, MLP -337.50% on Infiltration) mean the generated samples are easier than held-out clean test samples, which is the opposite of what an adversarial test set should look like. The paper explicitly acknowledges this in Section 6: future work should evaluate PDR against an independent generator or a standard attack method such as FGSM or PGD rather than each model's own generator. The 100% PDR of all unaugmented models is also consistent with those models never having seen the BGAN's particular boundary distribution, rather than with the attack being intrinsically strong. Consequently, the headline claim that BGAN augmentation converts a 100% performance drop into negative PDR values is not established by the current evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dual-purpose BGAN for intrusion detection on CICIDS2017: the generator augments minority-class training samples (Botnet, Web_Attack, Infiltration) and also generates 'adversarial samples' used to compute a Performance Drop Rate (PDR). The authors report that BGAN augmentation improves TabTransformer Macro-F1 from 82.96% to 86.50%, and they claim that all non-augmented models collapse under their adversarial test (PDR=100%) while all BGAN-augmented models achieve negative PDR values, indicating improved robustness. Additional results include False Triggered Rate (FTR) under benign Gaussian noise. The clean-data classification results are plausible, but the adversarial-robustness claim is the main advertised contribution.","tokens_in":12015,"tokens_out":11538,"duration_ms":127429,"significance":"If the robustness claim were valid, the dual-purpose BGAN framework would be a useful contribution to adversarial intrusion detection. The paper is clearly organized and honestly documents some limitations, notably the failure to model the Infiltration class and the explicit future-work note that PDR should be evaluated against an independent generator or standard attacks such as FGSM/PGD. However, the central robustness result is not supported: the PDR evaluation is circular because the same BGAN both augments the training set and generates the 'adversarial' test samples. Negative PDR values indicate that the generated samples are easier than clean held-out test samples, which is the opposite of an adversarial test. The clean-data contribution is modest and below several existing results, and the proposed model is outperformed by Random Forest and Decision Tree on clean data; the paper's value therefore rests almost entirely on the invalid robustness evaluation.","major_comments":[{"comment":"The central robustness result rests on a circular evaluation. Section 3.2 states that the trained BGAN is used both to augment the training set and to generate adversarial samples; Section 3.5 computes PDR (Eq. 8) on those BGAN-generated samples. For every Full-BGAN model, the attack distribution is therefore essentially a subset of the training distribution, so high F1_adversarial is expected regardless of robustness. The negative PDR values in Table 6 mean the generated samples are easier than the clean test set, the opposite of an adversarial test. Section 6 explicitly acknowledges this: future work should evaluate PDR against an independent generator or standard attacks such as FGSM/PGD. This invalidates the headline claim that BGAN augmentation converts 100% PDR into negative PDR.","section":"§3.2, §3.5, Table 6"},{"comment":"The evaluation does not use standard adversarial examples. 'Adversarial samples' here are unconstrained outputs of a generative model, not small perturbations of real inputs produced by an attack algorithm. A 100% PDR on all unaugmented models is consistent with simple distribution shift: those models never saw the BGAN's output distribution. Under Eq. (8), a negative PDR is interpreted as 'improved resilience,' but it literally means F1 is higher on the generated set than on the held-out clean set. This is not a valid robustness measure. The paper needs to report PDR under an input-specific attack with a bounded perturbation budget (or at minimum a genuinely independent generator), and should not interpret negative values as a sign of adversarial robustness.","section":"§3.5, Eq. (8)"},{"comment":"Even under the paper's own metric, the central summary is factually inconsistent. The abstract claims 'all BGAN-augmented models achieved negative PDR values,' and Section 4.3 says 'every model except Mixed on Infiltration reaches negative PDR.' But Table 6 reports Decision Tree Full BGAN Infiltration PDR = 0.00%, and the Mixed scenario has 100% on Infiltration. The claim is therefore false as stated. Moreover, MLP Full BGAN reaches −337.50% on Infiltration, a value the authors themselves attribute to memorization of the generator's distributional signature (§4.3); using such values as evidence of robustness is contradictory.","section":"Table 6, §4.3, Abstract"}],"minor_comments":[{"comment":"Infiltration F1 decreases from 0.8571 (Baseline) to 0.7500 (Full BGAN); the narrative that BGAN improves all targeted minority classes should be softened.","section":"Table 5"},{"comment":"The Wasserstein-distance quality thresholds ('<1.5 OK; ≥3.0 BAD') are introduced without justification, and the Gaussian fallback for Infiltration is not a faithful substitute for the class distribution.","section":"§3.2, Table 2"},{"comment":"The Mixed scenario's FTR anomaly at σ=0.05 (9.28%) is attributed to the Gaussian fallback using the same σ; this confound should be analyzed more carefully.","section":"Table 7"},{"comment":"No confidence intervals or multiple seeds are reported; all accuracy/F1 comparisons rely on a single stratified split. BGAN training hyperparameters (learning rate, batch size, early stopping criteria) are only partially specified, which limits reproducibility.","section":"§3, §4"},{"comment":"The figure labels BGAN outputs as 'Adversarial Samples.' This is misleading: they are generated samples, not adversarially perturbed inputs.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline reject/major-revision case. I chose major_revision because the flaw is localized to the robustness evaluation and the authors' own future-work paragraph identifies the correct remedy. However, the revised manuscript must redo the PDR evaluation with standard adversarial attacks (e.g., FGSM/PGD) or a genuinely independent generator, and must not interpret negative PDR as a sign of robustness. If the authors cannot provide such evidence, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the headline result here is not established. The paper's robustness evaluation uses adversarial samples generated by the very same BGAN that augmented the training set. Table 6's PDR therefore measures how well the classifier recognizes its training generator's outputs, not its resilience to adversarial perturbation. The negative PDR values, where models score higher on 'adversarial' samples than on clean test data, are the tell: those generated samples are easier than held-out benign flows. The paper's own future-work section concedes that PDR against an independent generator or standard attacks (FGSM, PGD) is needed. So the central contribution—'BGAN consistently enhances adversarial robustness'—is unsupported.\n\nWhat is actually new: the dual-purpose framing of a single BGAN for minority-class augmentation and attack-sample generation is a combination I haven't seen, though both pieces exist separately ([1] does BGAN augmentation, [16] does TabTransformer for IDS). The clean-data Macro-F1 improvement from 82.96% to 86.50% on CICIDS2017 is plausible, and the Web_Attack jump (0.29→0.61) is the kind of result augmentation should produce. The paper is honestly written: it reports the Infiltration BGAN failure, uses Gaussian fallback, and flags its own limitations. The FTR analysis under Gaussian noise is fine as a stability check, though it is not an adversarial evaluation.\n\nSoft spots beyond the circularity: the baselines Random Forest and Decision Tree get higher clean Macro-F1 than TabTransformer, which the paper explains as a linear separability artifact, but that weakens the framing. The Mixed scenario's FTR anomaly at σ=0.05 is explained but shows the Gaussian fallback contaminates the robustness story. None of that matters as much as the load-bearing flaw. The negative PDR is the opposite of what an adversarial test set should produce.\n\nBottom line: a competent, clearly written empirical study marred by a fatal-to-the-main-claim evaluation design. The clean-data augmentation result could be salvaged, and the robustness question could be re-answered with FGSM/PGD or an independent generator. That's a reasonable revision path, not a repudiation of the whole paper. I'd send it to a serious referee—the topic is timely and the error is instructive—but the verdict should be reject in its current form. I would not cite the robustness numbers.","headline":"The BGAN-TabTransformer robustness claim is circular and unsupported; the clean-data Macro-F1 gain is real, but the paper's headline conclusion doesn't survive.","tokens_in":12529,"tokens_out":2533,"would_cite":false,"duration_ms":27728,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One boundary-seeking GAN both fixes class imbalance and produces adversarial probes, turning a 100% performance drop into negative drop rates for a TabTransformer classifier.","keywords":["intrusion detection","class imbalance","adversarial robustness","boundary-seeking GAN","TabTransformer","data augmentation","performance drop rate","network security"],"falsifier":"Re-run the augmented TabTransformer's PDR evaluation using adversarial samples from an independent generator or a standard attack method such as FGSM or PGD. If PDR turns positive or the negative values disappear, the claimed robustness gain was specific to the training generator; the paper's own future-work section specifies this test.","tokens_in":11563,"feed_emoji":"🛡️","tokens_out":5736,"duration_ms":55217,"temperature":0.7,"pith_summary":"The paper argues that a single generative model, a Boundary-Seeking GAN, can serve two roles at once: synthesizing rare attack-class samples to fix severe class imbalance, and producing boundary-adjacent adversarial samples to stress-test a classifier. On the CICIDS2017 flow dataset, augmenting a TabTransformer this way raises Macro-F1 from 82.96% to 86.50%, with the Web_Attack class improving from 0.29 to 0.61. The stronger claim is robustness: every model trained without augmentation scored a 100% Performance Drop Rate on the generated adversarial samples, while nearly every augmented model scored negative PDR, meaning it held or improved on those samples. The paper reads this as evidence that boundary-region exposure during training is what makes an intrusion detector resilient, and that a self-attention tabular model exploits that exposure better than trees or plain networks. A sympathetic reader would care because it offers one lever for two persistent IDS problems: imbalance and evasion.","feed_headline":"One GAN both fixes class imbalance and hardens intrusion detection","feed_subtitle":"A boundary-seeking generator lifts minority-class F1 and converts total attack collapse into measured gains.","key_machinery":"The load-bearing mechanism is the Boundary-Seeking GAN's importance weight, w(x) = D(x)/(1−D(x)), which upweights generated samples the discriminator finds ambiguous (D(x) ≈ 0.5), steering synthesis toward the class boundary rather than the bulk of the minority-class distribution. That boundary-focused objective is what lets one generator do double duty: it augments rare classes (Botnet, Web_Attack, Infiltration) and then produces the adversarial samples used to compute PDR. The classifier side is the TabTransformer, whose self-attention embeds each of the 78 continuous flow features in the context of all others; the paper argues this contextual representation is why TabTransformer turns bou","core_discovery":"On the paper's own terms, the central discovery is that a single Boundary-Seeking GAN, trained to emphasize samples near the benign/malicious decision boundary, can serve both as an augmentation engine for rare attack classes and as an adversarial probe for the same classifier. Trained on CICIDS2017, TabTransformer with full BGAN augmentation reaches 99.61% accuracy and 86.50% Macro-F1, up from 82.96% without augmentation, with the Web_Attack class improving from 0.29 to 0.61 F1. In the robustness evaluation, every model with no augmentation scored a 100% Performance Drop Rate on BGAN-generated samples, while essentially every augmented model scored negative PDR — TabTransformer at −24.75% f","pith_inferences":["Because the adversarial test samples come from the same BGAN that augmented the training set, the negative PDR values are best read as the model adapting to that generator's distribution; an independent attack generator or FGSM/PGD would be the stricter test the authors themselves flag as future work.","The dual-purpose design suggests a cheap closed-loop recipe — generate boundary samples, train, re-probe with the same generator — that could be extended to other tabular classifiers and other tabular security tasks such as fraud or malware detection.","A testable extension follows from the Infiltration failure: combine boundary-seeking generation with few-shot or transfer methods for classes with fewer than roughly 30 samples, then check whether PDR can move below 100% for those classes.","The FTR stability result implies that attention-based contextual embeddings may be intrinsically more noise-tolerant than feature-independent architectures; comparing PDR and FTR curves across architectures, as the paper begins to do, is a direct way to test that claim."],"forward_implications":["If the PDR results hold, exposure to boundary-region samples during training is sufficient to stop total collapse under similar adversarial inputs — no separate defense mechanism is needed.","The same BGAN can replace two components in an IDS pipeline: resampling for imbalance and an external attack generator for robustness checks.","Classes with heavy feature overlap, like Web_Attack, are where boundary-seeking augmentation pays off most; gains there drive most of the Macro-F1 improvement.","Extremely rare classes such as Infiltration (29 training samples) are a hard limit for this approach; the Gaussian fallback keeps training stable but leaves adversarial robustness for that class at 100% PDR.","Clean-data accuracy is not a proxy for adversarial resilience: Random Forest and Decision Tree score higher Macro-F1 but show worse robustness trade-offs after augmentation."],"fun_headline_variants":["BGAN augmentation boosts TabTransformer F1 and robustness","One GAN: oversamples rare attacks and hardens IDS","Boundary-seeking GAN lifts rare-class F1 and resists adversarial attacks","TabTransformer with BGAN gains 86.5% Macro-F1 and survives adversarial probes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the BGAN-generated samples used in the PDR test are genuinely adversarial; if they are just familiar inputs from the same generator used in training, then the 100%-to-negative PDR shift shows adaptation to one generator, not resistance to attack.","fun_headline_variants_meta":{"raw":{"variants":["BGAN augmentation boosts TabTransformer F1 and robustness","One GAN: oversamples rare attacks and hardens IDS","Boundary-seeking GAN lifts rare-class F1 and resists adversarial attacks","TabTransformer with BGAN gains 86.5% Macro-F1 and survives adversarial probes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3733,"prompt_tokens":801,"completion_tokens":2932,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2853}},"tokens_in":545,"tokens_out":2932,"duration_ms":20548,"temperature":1.0,"reasoning_tokens":2853,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:01:32.367802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the augmented TabTransformer's PDR evaluation using adversarial samples from an independent generator or a standard attack method such as FGSM or PGD. If PDR turns positive or the negative values disappear, the claimed robustness gain was specific to the training generator; the paper's own future-work section specifies this test.","supporting_citations":[],"review_version":1}