{"id":"fb264e72-1e13-42f2-bd54-c8e4ffb5aaab","arxiv_id":"2501.07044","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"PROTEGO is a proposed plug-in detector that classifies adversarial versus normal images for Vision Transformers using the difference between adversarial and clean token features, reporting AUC over 0.95.","lead":"This paper proposes PROTEGO, a detector that aims to tell adversarial images apart from normal ones for Vision Transformer models by looking at the model's internal attention features. The authors report very high detection accuracy (AUC above 0.95) on several attack types, but the way the detector is built appears to use information that would not be available when actually guarding a system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Protego's features_noise (Eq. 6) is the difference between adversarial and clean features of the same input, so evaluating the detector on paired clean/adversarial images leaks the label; the reported AUC is not a valid detection metric.","rationale":"The paper's central claim is that a linear detector on features_noise detects adversarial examples. The load-bearing assumption is that features_noise can be computed at test time. The paper never specifies this. Eq. 6 requires both adversarial and clean features of the same input, which is exactly the label information. The evaluation in Table IV must have used paired data because adversarial examples are generated from the clean validation set; otherwise features_noise for a clean image cannot be formed. Thus the reported AUC is not evidence for a deployable detector. This is a test-time information leak, not an internal inconsistency. The reader identified the same weakest assumption; I agree. The paper also has other issues (single baseline, no code, inconsistent norms in Table III), but the feature-leak is sufficient for rejection. No attempt at adaptive attacks or unseen attacks is made despite claims in Section II-C. Given that the central construction is invalid as a detector, the correct verdict is REJECT.","tokens_in":10384,"tokens_out":4242,"duration_ms":39091,"concrete_test":"Request the inference-time algorithm for computing features_noise for an unlabeled query. To test experimentally: re-run detection with features_noise replaced by D(x) − D_clean_ref, where D_clean_ref is the mean clean [CLS] feature from the training set (a fixed reference), and compare AUC against Table IV. If AUC drops below 0.95, the reported results depend on per-sample clean/adversarial pairing. Also check whether the evaluation protocol uses paired differences; if it does, the method is not a detector.","verdict_should_be":"REJECT","load_bearing_attack":"Section III-A defines features_noise = D_adv(x) − D_clean(x) (Eq. 6), where D_adv and D_clean are the [CLS]-token features of the adversarial and normal versions of x. The detector described in Section III-B is trained and evaluated on this feature. But at inference time, a query x arrives without a paired 'clean' version. If x is adversarial, D_clean(x) is the feature of the original, unperturbed image, which the detector does not know; if x is clean, D_adv(x) is undefined. The only way to compute Eq. 6 is to know whether x is adversarial (i.e., the ground-truth label). Table IV likely evaluates on the ImageNet-val set where each adversarial example has a known clean source, so the difference is computed with that pair. This makes the high AUC (including perfect 1.0) an artifact of label-paired features, not a measure of detection ability. The paper's own limitation statement (Section V-A) concedes 'this method is not applicable' in many cases. No alternative inference-time procedure is given, so the central claim fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PROTEGO, a detection framework for adversarial examples on Vision Transformers (ViTs). The method extracts [CLS]-token features from a chosen transformer layer, computes the difference between the features of adversarial and clean versions of the same input, and trains a one-layer linear classifier on this difference. Experiments on ImageNet-val with three ViT models and six attack types report AUC values above 0.95, including several perfect scores of 1.0. The paper also presents attention-rollout visualizations to motivate the approach.","tokens_in":10621,"tokens_out":2292,"duration_ms":22954,"significance":"If the proposed feature were computable at inference time without knowledge of the ground-truth label, the high AUC values would constitute a strong result for ViT security. The paper also contributes an interpretability analysis of adversarial examples via attention rollout. However, the central feature definition (Eq. 6) appears to require pairing each adversarial input with its clean source, which is precisely the label information a detector lacks at test time. The paper's own limitation statement in Section V-A concedes that 'this method is not applicable' in many cases, but no alternative inference-time procedure is provided. As a result, the reported detection performance is not a valid measure of adversarial-example detection ability.","major_comments":[{"comment":"The feature used by the detector is defined as features_noise = D_adv(x) - D_clean(x). At inference time, an input x arrives without a paired clean version. If x is adversarial, D_clean(x) is the feature of the original unperturbed image, which the detector does not know; if x is clean, D_adv(x) is undefined. The only way to compute this feature is to know whether x is adversarial, i.e., to have the ground-truth label. The evaluation in Section IV therefore appears to pair each adversarial example with its clean source on ImageNet-val, leaking the label into the feature and making the AUC in Table IV an artifact of this pairing rather than a valid detection metric. The paper's own limitation statement in Section V-A admits the method 'is not applicable' in many cases, but no alternative test-time procedure is given.","section":"Section III-A, Eq. (6)"},{"comment":"The experimental comparison is limited to a single baseline (LID), with no error bars, no multiple runs, and no statistical significance tests. The perfect AUC values (e.g., 1.0 for FGSM on ViT-B-16 and for PGD on DeiT-Tiny) are reported without any variance analysis, which is particularly concerning given that the detector is a single linear layer trained on a feature that appears to encode the label. The claim of 'superior performance than existing detection methods' is not supported by a comparison with only one baseline, and the baseline's parameters (number of neighbors set to 10) are not justified.","section":"Section IV-C, Table IV"},{"comment":"The limitation statement that feature subtraction 'in many cases, this method is not applicable' directly contradicts the paper's framing of PROTEGO as a 'universal' and 'plug-in' detector (contribution list in Section I). This is not merely a scope limitation; it reflects the fundamental requirement of paired clean-adversarial data, which is unavailable in any realistic deployment scenario. The discussion does not propose a remedy, so the central detection claim is not defensible.","section":"Section V-A"},{"comment":"The paper claims that 'our method can effectively withstand adaptive attacks and defend against previously unseen attack methods,' but no adaptive attacks are evaluated and the only unseen attack considered is Patch-fool, which is still within the same threat model of known perturbation methods. This claim is unsupported by the experiments in Section IV.","section":"Section II-C"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical and grammatical errors, for example 'performace' (Section IV-A), 'The Protego is a charm' (Figure 1 caption), and inconsistent use of 'self-guard' versus 'PROTEGO' (Section I). These should be corrected.","section":"Throughout"},{"comment":"The text states that the comparison includes 'feature squeezing(FS) and local intrinsic dimensionality(LID),' but Table IV only reports LID and Ours; feature squeezing results are missing.","section":"Section IV-C"},{"comment":"The choice of the feature extraction layer (b1 from the encoder output) is not justified experimentally. The paper does not report results for features extracted from different layers, even though this is a free parameter that could affect detection performance.","section":"Section III-A"},{"comment":"The paper does not provide details on the size of the training set for the detector, the split between training and test data, or the number of samples per attack. This impairs reproducibility.","section":"Section IV-A"},{"comment":"The paper acknowledges lack of transferability between models and datasets but does not report any transfer experiments; this is listed as a limitation but the absence of quantitative evidence is not discussed in the main evaluation.","section":"Section V"}],"recommendation":"reject","confidential_remarks":"The core circularity in Eq. (6) is a load-bearing flaw that invalidates the main empirical claims. The paper also has a very weak experimental comparison (one baseline, no variance) and an internal contradiction between the claimed universality and the admitted inapplicability. The attention-rollout visualizations and the attack success measurements in Tables II and III are potentially useful as descriptive results, but they do not salvage the detector claim. I see no straightforward fix within the scope of the manuscript as written; re-framing the contribution as a study of feature differences between paired clean and adversarial examples, rather than a deployable detector, would require major revision and new experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: the detector in this paper is trained and evaluated on a feature that requires knowing the ground-truth label. Eq. (6) defines features_noise = D_adv(x) - D_clean(x), the difference between the [CLS] token features of the adversarial and clean versions of the same input. At inference time you don't have that pair. If the input is adversarial, its clean original is unknown; if it's clean, there is no adversarial counterpart. So the only way to compute that feature is to already know whether the sample is adversarial. The reported AUCs (mostly above 0.95, often 1.0) are an artifact of the evaluation protocol, not a measure of detection skill. The paper's own Section V-A admits 'in many cases, this method is not applicable,' which is a direct statement of the problem.\n\nThat's the load-bearing flaw, and it breaks the central claim. I want to give credit where due: the paper does provide a systematic attack evaluation of six attacks on three ViT models, with attention rollout and gradient rollout visualizations that show a difference between clean and adversarial inputs. That empirical part is fine, and the idea of using transformer internal features for detection is reasonable, though it is a routine extension of existing feature-based detectors (LID, KD, BU) applied to the [CLS] token. The comparison, however, is against only one baseline (LID), with no error bars, no multiple runs, and no code. Some perturbation norms in Table III look inconsistent (PGD and FGSM L-infinity ~2.06, which doesn't match the epsilon=0.0625 in Table I), and that inconsistency is never discussed.\n\nIf the authors can't supply an inference-time procedure that doesn't require the label, the method cannot be used as a detector. That's not a minor weakness; it is a complete mismatch between the reported experiment and the stated purpose. Even the discussion section's proposed fix (feature engineering) doesn't address the fundamental issue.\n\nWho is this for? Maybe someone collecting examples of label leakage in adversarial detection papers. It doesn't deserve serious referee time in its current form; the correct response is to go back and redesign the feature so that it is computable from the test input alone and then rerun the evaluation. Until then, the empirical claims should not be taken at face value.\n\nMy recommendation: treat this as a clear reject. If you want a cautionary example of why you must check whether the detector's input is actually available at test time, it's useful, but not as a citation for a working detection method.\n\nAs always, my read of the stress-test note: it holds up exactly.","headline":"Protego's detector feature requires knowing the ground-truth label (Eq. 6), so the reported AUCs are a label-leakage artifact, not valid detection performance.","tokens_in":11148,"tokens_out":2077,"would_cite":false,"duration_ms":19895,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes PROTEGO, a one-layer linear detector trained on the difference between adversarial and clean [CLS]-token features, and reports AUC above 0.95 for six attack types on three ViT models.","keywords":["adversarial example detection","Vision Transformer","CLS token features","attention rollout","plug-in detector","ImageNet","adversarial attacks"],"falsifier":"Use the same one-layer detector but replace each adversarial input's clean reference with the model's average clean [CLS] feature (or a clean feature from a different image), and compare the AUC; a large drop would show the reported detection relied on the paired clean-adversarial comparison rather than on a property intrinsic to adversarial examples.","tokens_in":10200,"feed_emoji":"🛡️","tokens_out":9628,"duration_ms":83051,"temperature":0.7,"pith_summary":"PROTEGO claims that a vision transformer's own [CLS] token, the learned representation that summarizes an entire image for classification, carries a detectable difference between adversarial and normal inputs. By subtracting the clean token features from the adversarial token features and training a single linear layer on that difference, the paper reports AUC scores above 0.95 for all six tested attacks (PGD, FGSM, BIM, CW, MIM, and Patch-fool) on three pre-trained ViT models (ViT-B-16, ViT-B-32, and DeiT-Tiny), with several settings reaching AUC 1.0. If correct, this gives ViT-based systems a cheap, plug-in detector that needs no classifier retraining and no external feature extractor, which matters as ViTs become the visual backbone for multimodal and metaverse applications.","feed_headline":"A single linear layer detects six adversarial attacks on ViTs","feed_subtitle":"A plug-in detector on [CLS]-token feature differences reports AUC above 0.95 for six attack types on three ViT models.","key_machinery":"The central object is the [CLS] token feature vector extracted from the final transformer encoder layer, which the paper treats as a summary of all input information. The key operation is features_noise = D_adv(x) - D_clean(x), the per-sample difference between adversarial and clean feature vectors, followed by a flattening step and a one-layer linear classifier trained with cross-entropy and SGD with momentum. Attention rollout and gradient attention rollout are used as interpretability tools to show that adversarial examples concentrate attention on image edges rather than object parts, motivating why the feature-space difference is learnable.","core_discovery":"The paper's central claim is that adversarial perturbations leave a consistent, learnable trace in the [CLS] token features, isolated by Eq. (6) as features_noise = D_adv(x) - D_clean(x), and that a one-layer linear network trained on this difference separates adversarial from clean examples. In Table IV, AUC exceeds 0.95 in all eighteen model-attack combinations and reaches 1.0 in several cases, including FGSM on ViT-B-16 and PGD and FGSM on DeiT-Tiny. The paper further uses attention rollout and gradient attention rollout to argue that adversarial inputs shift the model's attention from semantically meaningful regions to image edges, which is presented as the intrinsic reason the feature difference exists.","pith_inferences":["The evaluation protocol matters for interpreting these results: if D_clean(x) in Eq. (6) comes from the same image that is then attacked, the detector is effectively handed the label, because the clean and adversarial features are paired per sample; a deployment-grade test would need a clean reference that is not the test image.","The paper's own discussion section concedes that the subtraction feature is not applicable in many cases; a natural extension would be to replace the per-sample clean reference with a model-level clean feature distribution and measure the resulting AUC drop.","A direct cross-model transfer experiment, training on one ViT's features and testing on another without retraining, would clarify whether the detected trace is intrinsic to adversarial examples or specific to each model's feature space.","Perfect AUCs in some settings raise the question of whether the detector separates the attack method rather than adversariality in general; training on five attack types and testing on the sixth would test that."],"forward_implications":["A plug-in detector can be added to any of the three ViT backbones without retraining the classifier or changing the image pipeline.","The high AUCs imply that a simple threshold on the detector's output can reject most adversarial inputs while keeping nearly all clean inputs for the evaluated attack settings.","If the attention-shift explanation is correct, defenses could be designed around the [CLS] token's feature geometry rather than around input preprocessing or classifier hardening.","The reported numbers are specific to ImageNet validation images, so applying PROTEGO elsewhere would require re-running the feature-difference training on that domain's data."],"supporting_citations":[{"why":"Supplies the ViT architecture and the [CLS] token from which the detector's features are extracted.","marker":"[6]"},{"why":"Defines the FGSM attack, one of the six attacks evaluated and the basis for iterative variants.","marker":"[15]"},{"why":"Defines the BIM attack, an iterative FGSM variant used in the evaluation.","marker":"[16]"},{"why":"Defines the PGD attack, a white-box iterative attack used in the evaluation.","marker":"[17]"},{"why":"Defines the momentum-based MIM attack, one of the six attacks used in the evaluation.","marker":"[18]"},{"why":"Defines the Carlini-Wagner attack, one of the six attacks used in the evaluation.","marker":"[31]"},{"why":"Proposes LID, the baseline detection method to which PROTEGO's AUC is compared in Table IV.","marker":"[21]"},{"why":"Provides attention rollout, used to visualize and motivate why adversarial attention regions differ from clean ones.","marker":"[29]"},{"why":"Provides gradient attention rollout, the class-specific visualization used to show adversarial attention shifts to image edges.","marker":"[30]"}],"fun_headline_variants":["One linear layer spots six adversarial attacks on ViTs","ViT defenses: Simple detector achieves AUC over 0.95 for six attacks","Protego: [CLS] token differences reveal adversarial inputs with 0.95 AUC","ViT adversarial detection: one layer, AUC over 0.95"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a real detector can obtain the clean feature D_clean(x) for the same input x whose adversarial version arrives at test time, because Eq. (6) needs both; without paired clean-adversarial features, the high reported AUCs may not transfer to deployment.","fun_headline_variants_meta":{"raw":{"variants":["One linear layer spots six adversarial attacks on ViTs","ViT defenses: Simple detector achieves AUC over 0.95 for six attacks","Protego: [CLS] token differences reveal adversarial inputs with 0.95 AUC","ViT adversarial detection: one layer, AUC over 0.95"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001422,"raw_usage":{"total_tokens":5730,"prompt_tokens":924,"completion_tokens":4806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":4724}},"tokens_in":540,"tokens_out":4806,"duration_ms":30597,"temperature":1.0,"reasoning_tokens":4724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:41.492700+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the same one-layer detector but replace each adversarial input's clean reference with the model's average clean [CLS] feature (or a clean feature from a different image), and compare the AUC; a large drop would show the reported detection relied on the paired clean-adversarial comparison rather than on a property intrinsic to adversarial examples.","supporting_citations":[{"cited_title":"J., & Bengio, S","cited_arxiv_id":null,"evidence_quote":"Defines the BIM attack, an iterative FGSM variant used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the momentum-based MIM attack, one of the six attacks used in the evaluation."},{"cited_title":"(2017, May)","cited_arxiv_id":null,"evidence_quote":"Defines the Carlini-Wagner attack, one of the six attacks used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides gradient attention rollout, the class-specific visualization used to show adversarial attention shifts to image edges."}],"review_version":1}