{"id":"2ef9a709-6dc7-4d7c-a7f3-0080f67ae69b","arxiv_id":"2412.18718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DETR object detectors are highly vulnerable to standard adversarial attacks, transfer attacks within the DETR family, and a new attack using intermediate losses cuts accuracy with smaller perturbations.","lead":"This paper tests how easily four versions of the DETR object detector, a transformer-based system used in self-driving research, are fooled by adversarial image attacks on COCO and KITTI, and introduces a new attack that combines several DETR losses. It matters because detection transformers are being used in safety-critical systems, so knowing their weak points under attack guides more robust deployment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing perturbation-size metrics make the proposed attack's 'SOTA with less visible perturbations' claim untestable, since PGD/C&W have explicit budgets while Ours has none.","rationale":"The reader's weakest assumption was the tuning protocol: grid-searched loss weights evaluated on the same COCO validation set, with no held-out split or error bars. That is a valid concern about out-of-sample generalization. However, I see an even more direct blocker for the paper's central novelty: the claim that the proposed attack achieves SOTA 'with less visible perturbations' is never supported by any quantitative perturbation metric. Table I contains only AP and AR; the only evidence for visibility is a pair of sample images. Meanwhile, PGD has an explicit L∞ budget and C&W is an L2-minimizing attack, while Algorithm 1 has no projection or stated norm bound. Without per-image norm statistics, the comparison between Ours and the baselines is not meaningful: lower AP can simply reflect a larger allowed perturbation. This concern also applies to the qualitative transferability discussion and to the comparison of perturbation levels in Sec. V-C. If the authors added perturbation metrics and matched perturbation budgets, the SOTA claim would become testable; if the numbers did not favor Ours, the claim would need to be weakened. This reinforces the reader's CONDITIONAL verdict rather than moving it: the robustness-of-DETR finding is plausible and consistent with prior work, but the headline contribution of the new attack needs explicit quantitative support. I also note a minor internal inconsistency in Algorithm 1: line 11 returns when 'Losstotal does not converge,' which is likely a typo for 'converges,' but further complicates independent reproduction until code is released.","tokens_in":12555,"tokens_out":5386,"duration_ms":48911,"concrete_test":"Run all attacks in Table I on the COCO validation set and report per-image median and mean L∞, L2, and LPIPS (AlexNet) perturbation norms, together with AP. Then match perturbation budgets: for each model, choose PGD ε such that its median L∞ equals that of Ours, and choose C&W c to match Ours's median L2, and compare AP at equal visibility. If Ours does not achieve lower AP than PGD/C&W at matched perturbation norms, or if its perturbations are not statistically smaller at equal AP, the 'SOTA with less visible perturbations' claim fails. As a secondary check, recompute Ours with ω selected on a held-out subset of COCO and report bootstrap 95% confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proposed attack's headline claim — 'achieves SOTA performance with less visible perturbations on the COCO dataset' (Sec. I and Sec. V-B.1) — is not testable from the reported data because perturbation size is never quantitatively measured. Table I reports only AP/AR; the only support for 'less visible' is two sample images (Fig. 3/4). Baseline C&W minimizes L2 while PGD uses a hard L∞ bound of ±10/255, whereas Algorithm 1 has no projection or stated norm constraint (α=0.3, c=0.8). Thus the comparison is not apples-to-apples: PGD's AP at ε=0.1 on R50-DC5 (0.073) is worse than Ours (0.047), but without per-image L∞/L2/LPIPS numbers one cannot tell whether this reflects greater attack strength or simply a larger perturbation budget. In fact, on R50 and R101, PGD ε=0.1 already outperforms Ours (0.070 vs 0.084, 0.060 vs 0.063), so the SOTA claim is model-dependent even on the reported AP axis. The grid-search over ω on the same validation set (Sec. IV-C, Table I) further prevents out-of-sample generalization, but the more fundamental blocker is the missing perturbation metric: even with a perfect tuning protocol, no quantitative comparison with PGD/C&W would be possible. The central novelty of the paper therefore rests on an unverified quantity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical evaluation of the adversarial robustness of four DETR object-detection models (DETR-R50, DETR-R50-DC5, DETR-R101, DETR-R101-DC5) on MS-COCO and KITTI under three standard white-box attacks (FGSM, PGD, C&W) extended to DETR, plus a proposed two-stage attack that combines an initial one-step perturbation with a modified C&W procedure using intermediate decoder losses and a weighted total loss. It also reports intra-network transferability among DETR variants and cross-network transferability to Faster R-CNN, and it analyzes self-attention feature maps. The paper's central claims are that DETRs are significantly vulnerable to standard adversarial attacks and that the proposed attack achieves state-of-the-art performance with less visible perturbations on COCO.","tokens_in":12875,"tokens_out":10176,"duration_ms":82092,"significance":"If the vulnerability results hold, this is a useful reference point for the community: it extends well-known attacks to DETR, provides transferability data across DETR variants and to a CNN detector, and examines self-attention behavior under attack. The large AP drops under PGD and C&W in Table I provide robust support for the core vulnerability claim. However, the proposed attack's 'SOTA with less visible perturbations' claim is not supported by the current evidence, because perturbation size is never quantified, the loss weights are tuned on the same validation set used for evaluation, and no error bars are reported. The paper's main empirical contribution, the vulnerability benchmark, is plausible but needs the tuning and reporting issues addressed; the attack contribution needs substantial additional evidence. No code or machine-checked artifacts are provided at this stage.","major_comments":[{"comment":"The claim that the proposed attack 'achieves SOTA performance with less visible perturbations on the COCO dataset' is not testable from the reported data. Table I reports only AP and AR; no L2, L-infinity, or perceptual perturbation-size metric is given for any attack, and Algorithm 1 has no explicit norm budget or projection. Moreover, Eq. (10) defines Loss_dm between x_adv and x where x has already been modified by the initial one-step perturbation in Algorithm 1 (line 2), so the method does not directly minimize distance to the original clean image. The comparison is therefore not apples-to-apples with PGD (which has an L-infinity constraint, although its setup is ambiguous: the text states eps=0.03/0.1 while also specifying L-infinity bounds of +/-10/255) or with C&W (which minimizes L2). Even on the AP axis alone, the SOTA claim is model-dependent: PGD at eps=0.1 achieves lower AP than Ours on DETR-R50 (0.070 vs 0.084) and DETR-R101 (0.060 vs 0.063), while Ours is better only on the two DC5 variants (0.047 vs 0.073 and 0.034 vs 0.063). Quantitative perturbation-size measurements are required before this claim can be assessed.","section":"Sec. V-B.1, Table I, Algorithm 1, Eq. (10)"},{"comment":"The optimal loss weights omega in Eq. (9) are determined by a grid search on the same COCO validation set that is then used to report the attack comparison in Table I, as stated in Sec. IV-C ('The optimal weight values omega are determined using a grid search approach'). No held-out tuning set, grid range, or final weight values are provided. In addition, all AP/AR numbers are single-run and no error bars or confidence intervals are given, so the differences between Ours and PGD/C&W (for example, 0.047 vs 0.073 for DETR-R50-DC5) cannot be distinguished from run-to-run variation. An out-of-sample evaluation protocol, with standard deviations across at least a few runs, is needed to support the SOTA claim.","section":"Sec. IV-C and Table I"},{"comment":"The transferability metric TR in Eq. (7) is not a rate and is not bounded: TR_{m,n} = (AP^m_clean - AP^m_adv(n)) / (AP^n_clean - AP^n_adv(n)) can exceed 100% (e.g., PGD from DETR-R50 to DETR-R50-DC5 in Table II is 109.0%) and can be negative or undefined when the denominator is small. Because it is normalized by the source model's own AP drop, it conflates the source model's vulnerability with the transferability of the perturbations; the reported values may overstate intra-network transferability and understate cross-network transferability. The conclusions that 'PGD demonstrates the best intra-network transferability' and 'FGSM shows the best cross-network transferability' therefore need to be re-derived from raw AP values or from a properly defined transferability measure, such as the ratio of target AP drop at a matched perturbation budget.","section":"Eq. (7) and Table II"}],"minor_comments":[{"comment":"The FGSM formulation is ambiguous: Eq. (4) writes x_adv = x + eps sign(grad_x(-J(θ,x,t_c))) with J defined as a cross-entropy loss, which would decrease the loss if read literally; the accompanying sentence 'we set J = -J' is confusing. Please clarify the sign convention. Also, 'untargated' should be 'untargeted'.","section":"Eq. (4), Sec. IV-A"},{"comment":"Algorithm 1 has several reproducibility issues: line 10 states 'Update w_i with gradient decent w_i <- grad_{w_i} (...)' but omits the optimizer, learning rate, and iteration count for the inner update; line 11 'if Losstotal does not converge then return' appears inverted, since the method should return when it converges. Please provide the full optimization details and correct the convergence condition.","section":"Sec. IV-C, Algorithm 1"},{"comment":"Eq. (1) has a bracketing error: it reads [softmax(P(θ,x), sigmoid(B(θ,x)] and should be [softmax(P(θ,x)), sigmoid(B(θ,x))]. Similarly, Eq. (3) contains a typo ('ta denotes and ground-truth annotations').","section":"Eq. (1)"},{"comment":"The PGD setup is ambiguous: the text says 'eps is set to 0.03, and 0.1, with a total of 10 iterations' and then 'The L-infinity bounds are set to +/-10/255.' If eps is the step size, the total perturbation budget after 10 iterations is not stated; if eps is the L-infinity bound, the 10/255 statement is inconsistent with eps=0.1. Please report the exact budget and step-size schedule.","section":"Sec. V-A.1"},{"comment":"The qualitative claim that the proposed attack produces 'the least salient features' in the self-attention maps (Sec. V-C, Fig. 3) is based on a single sample image and should be supported by quantitative measures or by multiple examples.","section":"Sec. V-C / Fig. 3"},{"comment":"The paper states that code will be available upon publication, but for a benchmark-style evaluation paper, providing the code and configuration files in the initial submission would substantially improve reproducibility. Also, the 'first comprehensive study' claim in Sec. II should be softened given the prior work on DETR robustness cited in [23]-[25].","section":"General"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection because the core vulnerability finding is plausible and useful, but the SOTA claim for the proposed attack rests on missing perturbation metrics and in-sample tuning. The authors should either substantiate that claim with quantitative perturbation-size measurements, a proper held-out tuning protocol, and error bars, or remove it from the abstract and conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper's core empirical finding—DETR variants lose a lot of AP under FGSM/PGD/C&W—holds up and is useful. The headline claim for their own attack, 'SOTA performance with less visible perturbations,' does not hold as reported.\n\nWhat is actually new: this is the first systematic white-box and black-box robustness evaluation of multiple DETR variants (R50, R50-DC5, R101, R101-DC5) on both COCO and KITTI. That is worth having. The transferability analysis—high intra-family transfer, limited transfer to Faster R-CNN—is a solid addition. The self-attention map visualizations are a nice qualitative touch; they do show attention breaking down under attack.\n\nThe soft spots are real. Most importantly, 'less visible perturbations' is never measured. No L2, L∞, or LPIPS numbers anywhere in the paper. Algorithm 1 has no projection or explicit norm constraint, so the comparison with PGD (ε=0.1, L∞ bound ±10/255) and C&W (L2 minimization) is not apples-to-apples. Table I actually shows PGD at ε=0.1 beating their attack on two of four COCO models (AP 0.070 vs 0.084 on R50; 0.060 vs 0.063 on R101), so even on the AP axis the 'SOTA' claim is model-dependent. The stress-test note is right: the central novelty rests on an unverified quantity.\n\nThe tuning protocol is also a problem. The loss weights ω are grid-searched on the same COCO validation set where they later report SOTA (Sec. IV-C). That is selection on the evaluation set. Add single runs with no error bars, and the SOTA comparison with PGD and C&W is not out-of-sample. The transferability metric TR can exceed 100% (e.g., 109% in Table II), and the paper never discusses what that means. Minor issue: KITTI models are retrained for only 25 epochs rather than using official checkpoints, which limits comparability with published DETR results.\n\nWho this is for: researchers working on adversarial robustness of object detectors and anyone deploying DETR in safety-critical settings. The vulnerability survey part deserves attention; the proposed attack needs re-evaluation with proper perturbation-size reporting, held-out tuning, and ideally code.\n\nMy take: send it to peer review—the benchmark portion merits referee time—but the attack section needs major revision before it can be trusted. If the authors add perturbation metrics and redo the tuning honestly, it could be a solid contribution. As it stands, treat the 'less visible perturbations' claim as unverified.","headline":"The vulnerability survey of DETR variants is solid, but the proposed attack's 'SOTA with less visible perturbations' claim is unsupported because perturbation size is never measured.","tokens_in":13388,"tokens_out":2247,"would_cite":true,"duration_ms":21135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Detection Transformers are substantially vulnerable to standard white-box adversarial attacks, and an attack that uses all decoder-layer losses breaks them with smaller perturbations.","keywords":["adversarial robustness","DETR","object detection","white-box attacks","black-box transferability","FGSM","PGD","C&W attack"],"falsifier":"Re-run the proposed attack on COCO with the weights in Eq. (9) selected on a separate tuning split and compare AP at matched perturbation budgets (e.g., equal mean $L_2$ distortion or equal $L_\\infty$ bound) against PGD and C&W; if PGD or C&W then matches or beats the proposed attack's AP at equal visibility, the paper's 'SOTA performance' claim is falsified. A minimal version: report the mean and standard deviation of AP over several attack runs; if the gap to PGD is within run-to-run noise, the claimed improvement is not established.","tokens_in":12352,"feed_emoji":"🎯","tokens_out":6630,"duration_ms":55059,"temperature":0.7,"pith_summary":"This paper asks whether Detection Transformers (DETR) are robust to adversarial images, and finds that they are not: four DETR variants with ResNet-50 and ResNet-101 backbones lose a large share of average precision under fast gradient sign (FGSM), projected gradient descent (PGD), and Carlini-Wagner (C&W) attacks on both COCO and KITTI. The result matters because DETR is positioned for safety-critical perception in autonomous driving, where a small perturbation to an image could turn a detected pedestrian into a non-detection. To exploit DETR's structure, the authors propose an untargeted attack that first applies a one-step classification perturbation and then runs an iterative C&W-style optimizer that attacks classification, bounding-box regression, and GIoU losses from every decoder layer. On COCO this attack achieves the largest average-precision drop among the tested attacks at qualitatively smaller perturbations than PGD, and its success is reflected in a collapse of the self-attention feature maps. The paper also shows that adversarial examples transfer well among DETR variants but poorly to the CNN-based Faster R-CNN, suggesting that mixed-model ensembles could blunt black-box attacks.","feed_headline":"Standard attacks gut DETR detectors; new attack shrinks the noise","feed_subtitle":"The attention mechanism does not shield DETR; the new attack beats PGD and C&W at smaller perturbations.","key_machinery":"The load-bearing machinery is the two-stage attack in Algorithm 1. It begins with a one-step FGSM-style update on the classification loss and then optimizes a weighted sum $L_{\\mathrm{total}} = \\omega_1 L_2(x_{\\mathrm{adv}},x) + \\omega_2 L_{\\mathrm{cls}} + \\omega_3 L_{\\mathrm{bb}} + \\omega_4 L_{\\mathrm{iou}}$, where the last two terms are negative $L_1$ and GIoU losses computed from the final output and from each intermediate decoder layer. This loss mixture is what lets the attack push DETR off its object predictions while staying close to the slightly perturbed starting image; the weights $\\omega$ are selected by grid search on the same validation set.","core_discovery":"The central claim is that DETR and its variants are significantly susceptible to standard white-box attacks, and that the proposed intermediate-loss attack is more perturbation-efficient than PGD while matching or exceeding its degradation. Concretely, on COCO, PGD with $\\epsilon=0.1$ drops DETR-R50 AP from 0.420 to 0.070, while the proposed attack at $c=0.8$ reaches AP 0.084 on the same model and AP 0.047 on DETR-R50-DC5, below PGD's 0.073 at that model. The authors attribute the attack's effectiveness to its use of the intermediate decoder hidden states: gradients from classification, $L_1$ box regression, and GIoU losses at every decoder layer are combined, so the perturbation disrupts not just the final predictions but the internal representation of objects. Self-attention feature maps from the last encoder layer shrink and lose their object-shaped focus under successful attacks, which the paper reads as evidence that the attention mechanism does not protect DETR against adversarial examples.","pith_inferences":["If the decoder-layer losses are the true source of the attack's power, then attaching the same loss stack to a CNN detector with multiple prediction heads should reproduce the effect; this test is not in the paper.","The transfer-rate metric in Eq. (7) is normalized by per-model clean AP, so values above 100% do not necessarily mean stronger attacks; reporting matched $L_2$ or $L_\\infty$ perturbation norms would sharpen the comparison.","Because the loss weights are selected on the validation set, the state-of-the-art claim may be optimistic; a held-out tuning split or cross-validation would tell whether the advantage over PGD and C&W generalizes.","The KITTI results show a different attack ranking than COCO, suggesting dataset-specific training and object scales matter; that variation is worth studying rather than treating the COCO result as universal."],"forward_implications":["DETR-based perception systems in safety-critical settings can be degraded by simple white-box attacks, so robustness cannot be assumed from the attention mechanism alone.","Because adversarial examples transfer strongly across DETR variants, an attacker with access to any one DETR model can mount effective black-box attacks on other DETR models.","Limited cross-network transfer to Faster R-CNN implies that ensembles mixing transformer and CNN detectors may increase black-box robustness.","Self-attention feature maps shrink under successful attacks, which offers a diagnostic signal for detecting adversarial inputs and for designing attention-aware defenses."],"supporting_citations":[{"why":"Defines the DETR architecture and the four variants evaluated in all experiments.","marker":"[5]"},{"why":"Provides the FGSM one-step attack that is extended to DETR.","marker":"[8]"},{"why":"Provides the PGD iterative attack that the paper extends and uses as its strongest baseline.","marker":"[20]"},{"why":"Supplies the C&W attack formulation, including the change-of-variables optimization reused in the proposed attack.","marker":"[22]"},{"why":"Shows PGD attack success against CNN object detectors, the comparison point for DETR's vulnerability.","marker":"[13]"},{"why":"Faster R-CNN is the CNN detector used to test cross-network transferability.","marker":"[26]"},{"why":"COCO is the general object detection dataset for the main white-box and transferability evaluations.","marker":"[29]"},{"why":"KITTI is the autonomous driving dataset used to test the attacks in a scenario-specific setting.","marker":"[30]"}],"fun_headline_variants":["DETR fails standard attacks; new attack needs less noise","Attention not a shield: DETR falls to simple attacks","New DETR attack beats PGD with smaller perturbations","Object detection transformers: easy prey for adversarial noise","DETR vulnerable: intermediate loss attack cuts perturbation size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art claim rests on loss weights $\\omega$ chosen by grid search and evaluated on the same COCO validation set with single-run AP values, so the reported advantage over PGD and C&W may not be an out-of-sample result.","fun_headline_variants_meta":{"raw":{"variants":["DETR fails standard attacks; new attack needs less noise","Attention not a shield: DETR falls to simple attacks","New DETR attack beats PGD with smaller perturbations","Object detection transformers: easy prey for adversarial noise","DETR vulnerable: intermediate loss attack cuts perturbation size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1491,"prompt_tokens":995,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":611,"tokens_out":496,"duration_ms":4739,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:32:11.375607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the proposed attack on COCO with the weights in Eq. (9) selected on a separate tuning split and compare AP at matched perturbation budgets (e.g., equal mean $L_2$ distortion or equal $L_\\infty$ bound) against PGD and C&W; if PGD or C&W then matches or beats the proposed attack's AP at equal visibility, the paper's 'SOTA performance' claim is falsified. A minimal version: report the mean and standard deviation of AP over several attack runs; if the gap to PGD is within run-to-run noise, the claimed improvement is not established.","supporting_citations":[{"cited_title":"Adversarial attacks on faster r-cnn object detector,","cited_arxiv_id":null,"evidence_quote":"Shows PGD attack success against CNN object detectors, the comparison point for DETR's vulnerability."}],"review_version":1}