REVIEW 3 major objections 3 minor 43 references
Multi-Task Consistency-based Detection of Adversarial Attacks
T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Adversarial perturbations can be caught by inconsistencies between the outputs of multiple vision tasks; the paper reports 99.9% ROC-AUC against PGD on BDD100k.
desk verdict Abstract-only: the idea is practical and the headline number is striking, but the selection procedure and attacker-model scope are exactly what a referee must check before believing 99.9%. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the consistency score, a metric that quantifies how much the outputs of two vision tasks on the same input diverge — for instance, comparing object-detection boxes against instance-segmentation masks on the same image. The paper also contributes a selection procedure that identifies which task pairs provide the strongest inconsistency signal, so the detector can monitor a small set of outputs rather than all combinations.
What would settle it
Take the detection scheme as described and evaluate it on BDD100k frames perturbed by a different attack family, such as C&W or an adaptive attack that explicitly preserves cross-task consistency, and compare the ROC-AUC with the reported 99.9%. If clean frames already produce a broad distribution of consistency scores overlapping the attacked-frame distribution, the near-perfect separation will not hold outside the original PGD setup.
Extended reading notes
Core claim
The central claim is that an adversarial perturbation, though nearly invisible in the input image, makes different vision tasks disagree with one another in a measurable way, and that this disagreement is substantially stronger or differently shaped than what clean images produce. The paper asserts that a consistency score over task outputs can therefore serve as a practical attack detector, with a reported 99.9% ROC-AUC on the BDD100k validation set under the PGD attacker model considered. In the authors' framing, the defense is efficient because it piggybacks on inference outputs the multi-task perception system already generates.
Load-bearing premise
The scheme assumes that attack-induced inconsistencies between task outputs are distinguishable from the inconsistencies that occur naturally on clean images, and that the task pairs chosen during development remain discriminative for attacks encountered at deployment.
Editorial extensions
If this is right
- A multi-task perception system can add adversarial-attack monitoring essentially for free, because the detector uses outputs the tasks already compute.
- The detector can run on every frame without extra forward passes through a separate model.
- Within the PGD attacker model considered, the reported 99.9% ROC-AUC means clean and attacked frames are almost perfectly separated on BDD100k.
- The consistency-score idea is not tied to the specific task pair in the example and can be applied to any two vision tasks with overlapping spatial or semantic output.
Reading between the lines
- If the inconsistency signal is generic across gradient-based attacks, the same score could flag other attack families without retraining, though the paper only demonstrates PGD.
- The pair-selection stage could overfit to PGD-related perturbation geometry; an attacker aware of which pairs are monitored might craft a perturbation that keeps those particular outputs consistent.
- A natural extension is to run the detector on clean frames with realistic inconsistencies such as blur, occlusion, and rare object shapes, to test whether the threshold at 99.9% leaves a usable false-positive margin in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The abstract claims a lightweight adversarial-attack detection scheme for autonomous driving visual systems, based on inconsistencies among the outputs of multiple vision tasks such as object detection and instance segmentation. It introduces a consistency score metric, a procedure to select the best model pairs, and reports a ROC-AUC of 99.9% for detecting PGD attacks on the BDD100k validation set. The core claim is that attack-induced cross-task inconsistencies can be distinguished from normal inconsistencies with very high accuracy, at low computational overhead.
Significance. If the 99.9% AUC result is reproducible and generalizes beyond the specific validation configuration, the approach would be a valuable contribution: it offers a defense that does not require retraining or heavy side computations, leveraging existing multi-task perception outputs. The idea of exploiting cross-task consistency is well motivated, and the reported performance is striking. However, the abstract alone does not supply enough information to evaluate whether the result is robust, unbiased, or transferable, which tempers the significance assessment.
major comments (3)
- [Abstract] The central quantitative claim, 'ROC-AUC performance of 99.9% detection within the considered attacker model,' is not supported by any experimental details: the abstract reports no attack parameters (e.g., PGD epsilon, iterations, step size), no model architectures, no number of task pairs, no data split, and no error bars or repeated runs. Without these, the 99.9% figure cannot be assessed as a stable estimate rather than a single favorable run.
- [Abstract] The statement that the authors 'designed an approach to select the best model pairs' raises a clear selection-bias concern: if the pair selection, or any threshold tuning, is performed on the same validation split that is later used to compute the reported AUC, the 99.9% number is an optimistically selected maximum over many choices, not an unbiased detection rate. The manuscript must clarify that model-pair selection and any decision thresholds were fixed on a separate development set and that the reported AUC is on a held-out test set.
- [Abstract] The abstract does not provide evidence that attack-induced inconsistencies are separable from ordinary inconsistencies present on clean images. Since the detection rule is based on inconsistency magnitudes, the claim of near-perfect detection requires that clean images exhibit substantially smaller consistency mismatches than attacked images. The paper should present the distribution of consistency scores for clean vs. attacked inputs, and report the false-positive rate at the operating point used for the AUC, to demonstrate that the method does not merely flag all uncertain inputs as attacks.
minor comments (3)
- [Abstract] The abstract mentions 'multiple vision tasks, e.g., object detection and instance segmentation' but does not specify which tasks were actually used in the evaluation; this should be stated explicitly.
- [Abstract] The phrase 'within the considered attacker model' is vague; it should be defined precisely, e.g., the specific PGD variant, the perturbation budget, and the set of victim models.
- [Abstract] The claim '99.9% detection' is a ROC-AUC value, not a detection accuracy at a single threshold; the distinction should be made clear to avoid overstatement.
Circularity Check
No significant circularity found: the abstract contains no derivation chain, equations, or self-citations that could reduce the claimed 99.9% AUC to a fitted input.
full rationale
This is an abstract-only review, so the paper provides no equations, no derivation chain, and no self-citations to analyze. The central empirical claim is that the proposed consistency-based defense achieves 99.9% ROC-AUC on BDD100k validation within the considered PGD attacker model. The abstract mentions a consistency score metric and an approach for selecting the best model pairs, but it does not describe how those components were fitted or evaluated. It is possible that model-pair selection was performed on the same validation split and attack family used for the final AUC, which would make the number an optimistically selected maximum rather than an unbiased estimate; however, the abstract gives no evidence of that, and the hard rules require quoting a specific reduction (e.g., Eq. X = Eq. Y by construction, or a fitted parameter renamed as a prediction). No such reduction can be exhibited from the available text. The phrase 'within the considered attacker model' is a scope limitation, not a circularity. The absence of detail about held-out attack families or clean-image separability is a missing-support or validity concern, not a circularity concern. Therefore, under the proportionality and evidence rules, the appropriate finding is no significant circularity with score 0.
Assumptions & free parameters
assumptions (2)
- domain assumption Adversarial perturbations produce measurable inconsistency between outputs of different vision tasks.
- domain assumption The consistency score and model-pair selection generalize from the development data to the attack scenario evaluated.
Cite this review
Pith. "Pith review of Multi-Task Consistency-based Detection of Adversarial Attacks." pith.science (2026). https://pith.science/paper/FEY5CRG2
@misc{pith2026260807750,
author = {Pith},
title = {Pith review of: Multi-Task Consistency-based Detection of Adversarial Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/FEY5CRG2}},
note = {Machine review of arXiv:2608.07750}
}
read the original abstract
Deep Neural Networks (DNNs) have found successful deployment in numerous vision perception systems. However, their susceptibility to adversarial attacks has prompted concerns regarding their practical applications, specifically in the context of autonomous driving. Existing defenses often suffer from cost inefficiency, rendering their deployment impractical for resource-constrained applications. In this work, we propose an efficient and effective adversarial attack detection scheme leveraging the multi-task perception within a complex vision system. Adversarial perturbations are detected by the inconsistencies between the inference outputs of multiple vision tasks, e.g., object detection and instance segmentation. To this end, we developed a consistency score metric to measure the inconsistency between vision tasks. Next, we designed an approach to select the best model pairs for detecting inconsistencies effectively. Finally, we evaluated our defense against PGD attacks across multiple vision models on the BDD100k validation dataset. The experimental results demonstrated that our defense achieved a ROC-AUC performance of 99.9% detection within the considered attacker model.
Figures
Figures from the paper (35 more)
Reference graph
Works this paper leans on
-
[1]
Object detection in 20 years: A survey,
Z. Zou, K. Chen, Z. Shi, Y . Guo, and J. Ye, “Object detection in 20 years: A survey,”Proceedings of the IEEE, vol. 111, no. 3, pp. 257–276, 2023
2023
-
[2]
Towards deep learning models resistant to adversarial attacks,
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” inInternational Conference on Learning Representations, 2018
2018
-
[3]
Adversarial objectness gradient attacks in real- time object detection systems,
K.-H. Chow, L. Liu, M. Loper, J. Bae, M. E. Gursoy, S. Truex, W. Wei, and Y . Wu, “Adversarial objectness gradient attacks in real- time object detection systems,” inIEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications. IEEE, 2020, pp. 263–272
work page 2020
-
[4]
Adversarial training for free!
A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!”Advances in neural information processing systems, vol. 32, 2019
work page 2019
-
[5]
C. Xiang, T. Wu, S. Dai, J. Petit, S. Jana, and P. Mittal, “{PatchCURE}: Improving certifiable robustness, model utility, and computation effi- ciency of adversarial patch defenses,” inUSENIX Security Symposium, 2024, pp. 3675–3692
work page 2024
-
[6]
PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier,
C. Xiang, S. Mahloujifar, and P. Mittal, “PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier,” inUSENIX Security Symposium, 2022, pp. 2065–2082
work page 2022
-
[7]
Compression to the rescue: Defending from adversarial attacks across modalities,
N. Das, M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau, “Compression to the rescue: Defending from adversarial attacks across modalities,” inACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2018
work page 2018
-
[8]
Detecting adversarial perturbations in multi-task perception,
M. Klingner, V . R. Kumar, S. Yogamani, A. Bär, and T. Fingscheidt, “Detecting adversarial perturbations in multi-task perception,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 13 050–13 057
work page 2022
Show all 43 references
-
[9]
Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,
D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Glaeser, F. Timm, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,”IEEE Transactions on Intelligent Transportation Syst...
2020
-
[10]
A survey on 3d object detection methods for autonomous driving applications,
E. Arnold, O. Y . Al-Jarrah, M. Dianati, S. Fallah, D. Oxtoby, and A. Mouzakitis, “A survey on 3d object detection methods for autonomous driving applications,”IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 10, pp. 3782–3795, 2019
2019
-
[11]
Joint 3d instance segmentation and object detection for autonomous driving,
D. Zhou, J. Fang, X. Song, L. Liu, J. Yin, Y . Dai, H. Li, and R. Yang, “Joint 3d instance segmentation and object detection for autonomous driving,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1839–1849
2020
-
[12]
Multi-task adversarial attack,
P. Guo, Y . Xu, B. Lin, and Y . Zhang, “Multi-task adversarial attack,” arXiv preprint arXiv:2011.09824, 2020
2011 arXiv
-
[13]
Real-time memory efficient multitask learning model for autonomous driving,
S. Miraliev, S. Abdigapporov, V . Kakani, and H. Kim, “Real-time memory efficient multitask learning model for autonomous driving,” IEEE Transactions on Intelligent Vehicles, 2023
2023
-
[14]
Multitask learning,
R. Caruana, “Multitask learning,”Machine learning, vol. 28, pp. 41–75, 1997
1997
-
[15]
Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,
I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” inIEEE conference on computer vision and pattern recognition, 2017, pp. 6129–6138
2017
-
[16]
Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification,
Y . Lu, A. Kumar, S. Zhai, Y . Cheng, T. Javidi, and R. Feris, “Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification,” inIEEE conference on computer vision and pattern recognition, 2017, pp. 5334–5343
2017
-
[17]
Adversarial examples for semantic segmentation and object detection,
C. Xie, J. Wang, Z. Zhang, Y . Zhou, L. Xie, and A. Yuille, “Adversarial examples for semantic segmentation and object detection,” inIEEE international conference on computer vision, 2017, pp. 1369–1378. 9
2017
-
[18]
Towards deep learning models resistant to adversarial attacks,
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017
2017 arXiv
-
[19]
A novel industrial intrusion detection method based on threshold-optimized cnn-bilstm-attention using roc curve,
M. Lan, J. Luo, S. Chai, R. Chai, C. Zhang, and B. Zhang, “A novel industrial intrusion detection method based on threshold-optimized cnn-bilstm-attention using roc curve,” inChinese Control Conference (CCC), 2020, pp. 7384–7389
2020
-
[20]
Adversarial robustness in multi-task learning: Promises and illusions,
S. Ghamizi, M. Cordy, M. Papadakis, and Y . Le Traon, “Adversarial robustness in multi-task learning: Promises and illusions,” inAAAI Conference on Artificial Intelligence, 2022, pp. 697–705
2022
-
[21]
Bdd100k Model Zoo,
T. E. Huang, “Bdd100k Model Zoo,” https://github.com/SysCV/ bdd100k-models, 2021
2021
-
[22]
MMDetection: Open mmlab detection toolbox and benchmark,
K. Chen, J. Wang, J. Pang, Y . Cao, Y . Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y . Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmar...
1906 arXiv
-
[23]
Adversarially-aware robust object detector,
Z. Dong, P. Wei, and L. Lin, “Adversarially-aware robust object detector,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 297–313
2022
-
[24]
A baseline for detecting misclassified and out-of-distribution examples in neural networks,
D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,”arXiv preprint arXiv:1610.02136, 2016
2016 arXiv
-
[25]
Detection based defense against adversarial examples from the steganalysis point of view,
J. Liu, W. Zhang, Y . Zhang, D. Hou, Y . Liu, H. Zha, and N. Yu, “Detection based defense against adversarial examples from the steganalysis point of view,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4825–4834
2019
-
[26]
Detecting adversarial examples from sensitivity inconsistency of spatial-transform domain,
J. Tian, J. Zhou, Y . Li, and J. Duan, “Detecting adversarial examples from sensitivity inconsistency of spatial-transform domain,” inAAAI Conference on Artificial Intelligence, 2021, pp. 9877–9885
2021
-
[27]
Dla: dense- layer-analysis for adversarial example detection,
P. Sperl, C.-Y . Kao, P. Chen, X. Lei, and K. Böttinger, “Dla: dense- layer-analysis for adversarial example detection,” inIEEE European Symposium on Security and Privacy. IEEE, 2020, pp. 198–215
2020
-
[28]
Using self- supervised learning can improve model robustness and uncertainty,
D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song, “Using self- supervised learning can improve model robustness and uncertainty,” Advances in neural information processing systems, vol. 32, 2019
2019
-
[29]
Towards deep learning models resistant to adversarial attacks,
A. M ˛ adry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”stat, vol. 1050, p. 9, 2018
2018
-
[30]
Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,
A. Kendall, Y . Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in IEEE conference on computer vision and pattern recognition, 2018, pp. 7482–7491
2018
-
[31]
Multitask learning strengthens adversarial robustness,
C. Mao, A. Gupta, V . Nitin, B. Ray, S. Song, J. Yang, and C. V ondrick, “Multitask learning strengthens adversarial robustness,” inEuropean Conference on Computer Vision. Springer, 2020, pp. 158–174
2020
-
[32]
Improved noise and attack robustness for semantic segmentation by using multi-task training with self-supervised depth estimation,
M. Klingner, A. Bar, and T. Fingscheidt, “Improved noise and attack robustness for semantic segmentation by using multi-task training with self-supervised depth estimation,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 320– 321
2020
-
[33]
Defending against adversarial attack towards deep neural networks via collaborative multi-task training,
D. Wang, C. Li, S. Wen, S. Nepal, and Y . Xiang, “Defending against adversarial attack towards deep neural networks via collaborative multi-task training,”IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 2, pp. 953–965, 2020
2020
-
[34]
Syndistnet: Self-supervised monocular fisheye cam- era distance estimation synergized with semantic segmentation for autonomous driving,
V . R. Kumar, M. Klingner, S. Yogamani, S. Milz, T. Fingscheidt, and P. Mader, “Syndistnet: Self-supervised monocular fisheye cam- era distance estimation synergized with semantic segmentation for autonomous driving,” inIEEE/CVF winter conference on applications of computer vi...
2021
-
[35]
Uninet: A unified scene understanding network and exploring multi-task relationships through the lens of adversarial attacks,
N. K. Gurulingan, E. Arani, and B. Zonooz, “Uninet: A unified scene understanding network and exploring multi-task relationships through the lens of adversarial attacks,” inIEEE/CVF International Conference on Computer Vision, 2021, pp. 2239–2248
2021
-
[36]
Multitask adversarial attack with dispersion amplification,
P. Haleta, D. Likhomanov, and O. Sokol, “Multitask adversarial attack with dispersion amplification,”EURASIP Journal on Information Security, vol. 2021, no. 1, p. 10, 2021
2021
-
[37]
A survey of robust adversarial training in pattern recognition: Fundamental, theory, and methodologies,
Z. Qian, K. Huang, Q.-F. Wang, and X.-Y . Zhang, “A survey of robust adversarial training in pattern recognition: Fundamental, theory, and methodologies,”Pattern Recognition, vol. 131, p. 108889, 2022
2022
-
[38]
Adversarial robustness vs. model compression, or both?
S. Ye, K. Xu, S. Liu, H. Cheng, J.-H. Lambrechts, H. Zhang, A. Zhou, K. Ma, Y . Wang, and X. Lin, “Adversarial robustness vs. model compression, or both?” inIEEE/CVF International Conference on Computer Vision, 2019, pp. 111–120
2019
-
[39]
When nas meets robustness: In search of robust architectures against adversarial attacks,
M. Guo, Y . Yang, R. Xu, Z. Liu, and D. Lin, “When nas meets robustness: In search of robust architectures against adversarial attacks,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 631–640
2020
-
[40]
Towards adversarially robust object detection,
H. Zhang and J. Wang, “Towards adversarially robust object detection,” inIEEE/CVF International Conference on Computer Vision, 2019, pp. 421–430
2019
-
[41]
Class-aware robust ad- versarial training for object detection,
P.-C. Chen, B.-H. Kung, and J.-C. Chen, “Class-aware robust ad- versarial training for object detection,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 420–10 429
2021
-
[42]
Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector,
S.-T. Chen, C. Cornelius, J. Martin, and D. H. Chau, “Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector,” in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proce...
2018
-
[43]
As the perturbation strength increases, it results in stronger impact on the target model and causes higher inconsistency between model pairs
At the time of testing, the alert raised by our defense was not propagated to the downstream automated driving tasks, but only logged 4 Object Detection (a) Distance 30m Instance Segmentation (b) Distance 30m (c) Distance 25m (d) Distance 25m (e) Distance 20m (f) Distance 20m ...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.