REVIEW 4 major objections 5 minor 38 references
Stealthy Multi-Task Adversarial Attacks
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A multi-task adversarial attack can degrade a single targeted task while every other task keeps or improves its performance, by optimizing a weighted loss with negative weights on the spared tasks.
desk verdict A useful problem definition and a plausible empirical trick, but the central feasibility guarantee is asserted rather than proved and the paper overclaims adversarial-training coverage. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the relaxed attack objective in Equation (2): maximize $\sum_{i=1}^{m} w_i L_i(x+\delta, y_i)$ over perturbations $\delta$ with $\|\delta\|_p \le \epsilon$, where $w_{i_t} > 0$ for the targeted task and $w_i < 0$ for every other task. The negative weights turn the preservation of non-targeted tasks into a penalty inside a single gradient-based attack loop, so a standard PGD or IFGSM step on this objective moves the perturbation in a direction that hurts the target and compensates the others. The automated weight-search procedure (Algorithm 1) updates each negative weight in proportion to how far that task's current loss exceeds its clean value, then shrinks the update step for tasks that already satisfy the constraint. The paper asserts, without proof, that adjusting these weights yields a solution feasible for the original constrained problem (Equation 1).
What would settle it
Run the automated weight search on a multi-task model where two tasks have strongly conflicting gradients (for instance, depth estimation and semantic segmentation on Cityscapes) under a larger perturbation budget, and record every non-targeted task's post-attack loss. If any non-targeted loss exceeds its clean value after Algorithm 1 completes its fixed number of steps, the feasibility guarantee is violated and stealthiness collapses.
Extended reading notes
Core claim
The paper's central claim is that a multi-task adversarial attack can be made selective and stealthy: the attacker chooses one targeted task, and the perturbation is required to push that task's loss up while every other task's loss stays at or below its clean value. The authors propose the Stealthy Multi-Task Adversarial Attack ($SMTA^{2}$), which encodes this objective as a constrained optimization problem and relaxes it to maximizing a weighted sum of task losses, with a positive weight on the targeted task and negative weights on all non-targeted tasks. They supply two ways to set those weights: manual tuning and an automated per-step update that increases the magnitude of a negative weight when the corresponding task's loss rises above its clean value and attenuates the update once the constraint is satisfied. The paper reports experiments across two multi-task benchmarks and two gradient-based attack algorithms showing that non-targeted tasks maintain or improve their performance while the targeted task is degraded, and that the automated weight search matches manual tuning in attack strength.
Load-bearing premise
Solving the relaxed weighted-loss objective with negative weights on non-targeted tasks always produces a perturbation under which those non-targeted tasks' losses stay at or below their clean values, and the automated weight-adjustment rule is guaranteed to reach such weights.
Editorial extensions
If this is right
- An attacker can sabotage a security-critical task in a multi-task perception system while leaving auxiliary tasks (e.g., depth, normal estimation) unchanged, making the attack invisible to task-level monitoring.
- The automated weight search removes the need for manual hyperparameter scanning, so stealthy attacks can be adapted per input at runtime.
- The framework provides a new evaluation criterion for multi-task attacks: attack strength is only meaningful if non-targeted tasks stay at or above their baseline performance.
- Because non-targeted tasks can even improve under attack, defenders cannot rely on overall multi-task loss or per-task monitoring to detect these perturbations.
Reading between the lines
- The feasibility guarantee for the relaxed problem is asserted but not proven; if a task's gradients conflict too strongly with the targeted task, the negative-weight penalty may not suffice, and the attack would collapse into a non-stealthy attack for that instance. A counterexample with three tasks where two share a strong gradient conflict would test this directly.
- The abstract states results on adversarially trained models, but the reported experiments only cover undefended models; measuring whether the negative-weight mechanism survives PGD-based adversarial training would clarify the practical threat window.
- The same negative-weight trick could be adapted to other constrained adversarial objectives, such as preserving perceptual similarity (a non-target 'task' being a perceptual quality metric) or satisfying fairness constraints during an attack.
- Scaling the approach to multi-task models with more than three tasks may require coupling the weight updates, since each added non-targeted task dilutes the influence of the target's positive weight.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SMTA^2, a framework for selective adversarial attacks on multi-task models. It formulates the objective as a constrained optimization problem (Eq. 1) in which the targeted task loss is maximized while each non-targeted task loss must not exceed its clean value, then relaxes this to a weighted-sum maximization with a positive weight on the targeted task and negative weights on non-targeted tasks (Eq. 2). The authors present a manual weight-tuning procedure and an automated procedure (Algorithm 1), and evaluate the framework on NYUv2 and Cityscapes using PGD L2, PGD L∞, and IFGSM attacks, attacking each task in turn. The central empirical claim is that the targeted task is degraded while non-targeted tasks remain at or above baseline performance, and that the automated weight search is comparable in effectiveness to manual tuning.
Significance. If the central claim holds, the paper makes a useful contribution: it introduces a clean evaluation criterion for selective multi-task attacks and demonstrates that negative-weight relaxation can in practice steer perturbations toward a single task. The experimental scope is broad, covering two datasets, three tasks, three attack variants, and two weight-search strategies, and the tables show a fairly consistent pattern of preserved non-targeted metrics. However, the manuscript's central feasibility guarantee for Eq. (2) is asserted without proof, the abstract and conclusion claim experiments on adversarially trained models that do not appear in the experiments, and all performance numbers are single point estimates with no uncertainty quantification. These gaps prevent full acceptance in the current form, but the contribution is potentially publishable after substantial revision.
major comments (4)
- [Methodology, Eq. (2)] The statement "we can guarantee to find a solution that is feasible to problem (1)" is unsupported. In nonconvex deep-network losses, a maximizer of the weighted sum with negative non-target weights is not guaranteed to satisfy the constraints Li(x+delta) <= Li(x) for i != it: the positive target term can dominate an increase in a non-targeted loss, and the feasible set may be empty for some inputs and budgets. Algorithm 1's update rule is a heuristic with no convergence analysis or monotonicity result, so it does not certify feasibility either. The same applies to the manual method, which is claimed to guarantee feasibility after finite trials. Please either provide a formal guarantee under explicit conditions or rephrase this as an empirical claim, and supplement it with per-sample feasibility statistics (for example, the fraction of test images on which every non-targeted constraint is satisfied).
- [Abstract and Experiments] The abstract states that experiments are performed "on both undefended and adversarially trained models," and the conclusion says the framework is validated "across various multi-task datasets and attack methods," but the experiments section contains no adversarially trained model. All reported results in Tables 1-6 appear to use undefended pretrained models. Please add experiments with adversarially trained models or remove the unsupported claim from the abstract and conclusion.
- [Tables 1-6] All results in Tables 1-6 are single point estimates with no error bars, confidence intervals, or statistical tests. Because the central claim is that non-targeted task performance is "strictly equal or better," the paper should demonstrate that this inequality is not an artifact of a single evaluation subset or random seed. At minimum, report bootstrap confidence intervals over test images or repeated attack runs, and ideally provide a paired test comparing attacked and clean non-targeted metrics.
- [Algorithm 1 hyperparameters] The automated weight-search method is said to work "with a proper step size and attenuation coefficient," but the paper never reports the values of lambda, alpha, or the number of attack steps S used in any experiment. These hyperparameters determine whether the non-targeted constraints are satisfied, and no sensitivity analysis is provided. Without this information, the reproducibility of the empirical feasibility result is in question, and the claim that the method is substantially more efficient than manual tuning is not fully supported.
minor comments (5)
- [Tables 1-6] The tables do not explicitly label which row corresponds to which targeted task; the underlining mentioned in the captions is not visible in the rendered text. Please add row labels such as "target = semantic segmentation" to make the mapping unambiguous.
- [Table 7] Table 7 is captioned "Example images" but contains only numerical results. Please either include the actual example images or retitle the table as a quantitative comparison.
- [Equation (2)] The sum notation in Eq. (2) is corrupted in the rendering ("mX i=1"); it should be typeset as a standard summation.
- [Related Work and Experiments] The paper claims to be the "first standard and baseline" for stealthy multi-task attacks, but it does not compare against existing multi-task attack methods such as MTA under a unified setup. A quantitative comparison with prior multi-task attack baselines would make the state-of-the-art claim credible.
- [References] The references for PGD and IFGSM are nonstandard: Chen and Wainwright 2015 is not the canonical PGD attack reference. Please cite Madry et al. 2018 for projected gradient descent and the original I-FGSM reference by Kurakin et al. 2016.
Circularity Check
No significant circularity: the stealthiness condition is the attack's stated objective and is empirically verified, not derived from itself or from a fitted parameter renamed as a prediction.
full rationale
The paper's chain is: formulate constrained SMTA (Eq. 1), relax it to a weighted-sum objective with negative non-target weights (Eq. 2), and then tune those weights manually or via Algorithm 1. None of these steps reduces to its own input by construction. The relaxed problem is explicitly a relaxation, not an equivalent reformulation, and the claim that weight adjustment can produce a solution feasible to Eq. 1 is an unproved algorithmic assertion, not an identity: a maximizer of Eq. 2 is not by definition feasible for Eq. 1, and the paper never defines feasibility as 'optimizing Eq. 2 with tuned weights.' Algorithm 1 is a feedback heuristic that increases the penalty on a non-target task when its loss exceeds baseline; the fact that Tables 1–6 show non-target metrics at or below baseline is an empirical check of that heuristic, not a relabeling of the tuning objective as an independent discovery. The targeted-task attack metrics are also compared against non-stealthy and single-task baselines, giving the central claim independent empirical content. The only overlapping-author citation (Guo et al. 2024) appears in a Related Work survey aside and is not load-bearing; no uniqueness theorem, prior-work ansatz, or fitted parameter is presented as a prediction. The unsupported feasibility guarantee and the lack of convergence analysis for Algorithm 1 are correctness and rigor concerns, not circularity. Since no reduction by construction or self-citation chain exists, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- initial weight w_i for non-targeted tasks =
not reported (negative magnitudes)
- step size lambda_i =
not reported
- attenuation coefficient alpha =
not reported
assumptions (3)
- domain assumption Solving the weighted-sum relaxation (Eq. 2) with appropriate weights yields a feasible point for the constrained problem (Eq. 1)
- domain assumption Non-targeted task performance can be preserved by minimizing their losses with negative weights during the attack
- standard math Attack algorithms PGD/IFGSM can effectively maximize the weighted loss
Cite this review
Pith. "Pith review of Stealthy Multi-Task Adversarial Attacks." pith.science (2026). https://pith.science/paper/5TUD7WAZ
@misc{pith2026241117936,
author = {Pith},
title = {Pith review of: Stealthy Multi-Task Adversarial Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/5TUD7WAZ}},
note = {Machine review of arXiv:2411.17936}
}
abstract
Deep neural networks are highly vulnerable to adversarial perturbations, raising serious safety concerns in the real-world systems. While prior work mainly explores single-task attacks or jointly degrading all tasks in multi-task models, practical scenarios often demand more selective and stealthy attack strategies. To address this challenge, we propose Stealthy Multi-Task Adversarial Attack (SMTA$^{2}$), a novel framework that selectively degrades a targeted task while strictly preserving the performance of non-targeted tasks. We formulate this objective as a constrained multi-objective optimization problem and design task-aware adversarial perturbations that maximize degradation on the targeted task without causing collateral damage on non-targeted tasks. To enhance practicality, we further introduce an automated loss-weight tuning strategy that dynamically balances attack and preservation objectives. Experiments on two multi-task benchmarks NYUv2 and Cityscapes demonstrate that SMTA$^{2}$ achieves strong attack performance on targeted tasks while maintaining non-targeted tasks intact on both undefended and adversarially trained models, establishing the first systematic framework for stealthy and selective multi-task attack framework.
Figures
Reference graph
Works this paper leans on
-
[1]
Andor, D.; Alberti, C.; Weiss, D.; Severyn, A.; Presta, A.; Ganchev, K.; Petrov, S.; and Collins, M. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042
work page Pith review arXiv 2016
-
[2]
Carlini, N.; Mishra, P.; Vaidya, T.; Zhang, Y.; Sherr, M.; Shields, C.; Wagner, D.; and Zhou, W. 2016. Hidden voice commands. In 25th USENIX security symposium (USENIX security 16), 513--530
2016
-
[3]
Carlini, N.; and Wagner, D. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), 39--57. Ieee
2017
-
[4]
Caruana, R. 1997. Multitask learning. Machine learning, 28: 41--75
1997
-
[5]
Chen, Y.; and Wainwright, M. J. 2015. Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees. arXiv preprint arXiv:1509.03025
arXiv 2015
-
[6]
Chen, Z.; Badrinarayanan, V.; Lee, C.-Y.; and Rabinovich, A. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, 794--803. PMLR
2018
-
[7]
Chen, Z.; Wang, C.; and Crandall, D. 2022. Semantically stealthy adversarial attacks against segmentation models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 4080--4089
work page 2022
-
[8]
Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; and Schiele, B. 2016. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3213--3223
2016
Show all 38 references
-
[9]
Crawshaw, M. 2020. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796
2020 arXiv
-
[10]
E.; Yu, D.; Deng, L.; and Acero, A
Dahl, G. E.; Yu, D.; Deng, L.; and Acero, A. 2011. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on audio, speech, and language processing, 20(1): 30--42
2011
-
[11]
W.; Sasdelli, M.; Rajasegaran, R.; and Campbell, D
Du, A.; Chen, B.; Chin, T.-J.; Law, Y. W.; Sasdelli, M.; Rajasegaran, R.; and Campbell, D. 2022. Physical adversarial attacks on an aerial imagery object detector. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1796--1806
2022
-
[12]
Evgeniou, T.; and Pontil, M. 2004. Regularized multi--task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 109--117
2004
-
[13]
Ghamizi, S.; Cordy, M.; Papadakis, M.; and Le Traon, Y. 2022. Adversarial robustness in multi-task learning: Promises and illusions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 697--705
2022
-
[14]
J.; Shlens, J.; and Szegedy, C
Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572
2014 arXiv
-
[15]
Guo, J.; Sun, H.; Qin, M.; Yu, H.; and Zhang, T. 2024. A Min-Max Optimization Framework for Multi-task Deep Neural Network Compression. In 2024 IEEE International Symposium on Circuits and Systems (ISCAS), 1--5. IEEE
2024
-
[16]
Guo, P.; Xu, Y.; Lin, B.; and Zhang, Y. 2020. Multi-task adversarial attack. arXiv preprint arXiv:2011.09824
2020 arXiv
-
[17]
He, K.; Zhang, J.; Yan, Y.; Xu, W.; Niu, C.; and Zhou, J. 2020. Contrastive zero-shot learning for cross-domain slot filling with adversarial attack. In Proceedings of the 28th International Conference on Computational Linguistics, 1461--1467
2020
-
[18]
E.; Mohamed, A.-r.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T
Hinton, G.; Deng, L.; Yu, D.; Dahl, G. E.; Mohamed, A.-r.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T. N.; et al. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazin...
2012
-
[19]
J.; and Bengio, S
Kurakin, A.; Goodfellow, I. J.; and Bengio, S. 2018. Adversarial examples in the physical world. In Artificial intelligence safety and security, 99--112. Chapman and Hall/CRC
2018
-
[20]
LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): 2278--2324
1998
-
[21]
J.; and Johns, E
Liu, S.; James, S.; Davison, A. J.; and Johns, E. 2022. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091
2022 arXiv
-
[22]
Liu, X.; Li, J.; Ma, J.; Sun, H.; Xu, Z.; Zhang, T.; and Yu, H. 2023. Deep transfer learning for intelligent vehicle perception: A survey. Green Energy and Intelligent Transportation, 100125
2023
-
[23]
Lu, J.; Sibai, H.; and Fabry, E. 2017. Adversarial examples that fool detectors. arXiv preprint arXiv:1712.02494
2017 arXiv
-
[24]
Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations
2018
-
[25]
Mao, C.; Chiquier, M.; Wang, H.; Yang, J.; and Vondrick, C. 2021. Adversarial attacks are reversible with natural supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 661--671
2021
-
[26]
Moosavi-Dezfooli, S.-M.; Fawzi, A.; Fawzi, O.; and Frossard, P. 2017. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1765--1773
2017
-
[27]
Nassif, R.; Vlaski, S.; Richard, C.; Chen, J.; and Sayed, A. H. 2020. Multitask learning over graphs: An approach for distributed, streaming machine learning. IEEE Signal Processing Magazine, 37(3): 14--25
2020
-
[28]
Nguyen, A.; Yosinski, J.; and Clune, J. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 427--436
2015
-
[29]
Poursaeed, O.; Katsman, I.; Gao, B.; and Belongie, S. 2018. Generative adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4422--4431
2018
-
[30]
Sener, O.; and Koltun, V. 2018. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31
2018
-
[31]
Silberman, N.; Hoiem, D.; Kohli, P.; and Fergus, R. 2012. Indoor segmentation and support inference from rgbd images. In Computer Vision--ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, 746--760. Springer
2012
-
[32]
Sun, H.; Fu, L.; Li, J.; Guo, Q.; Meng, Z.; Zhang, T.; Lin, Y.; and Yu, H. 2024. Defense against adversarial cloud attack on remote sensing salient object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 8345--8354
2024
-
[33]
Xu, L.; Li, J.; Lin, W.; Zhang, Y.; Ma, L.; Fang, Y.; and Yan, Y. 2016. Multi-task rank learning for image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 27(9): 1833--1843
2016
-
[34]
Yu, J.; Dai, Y.; Liu, X.; Huang, J.; Shen, Y.; Zhang, K.; Zhou, R.; Adhikarla, E.; Ye, W.; Liu, Y.; et al. 2024. Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras. arXiv preprint arXiv:2404.18961
2024 arXiv
-
[35]
Zhang, T.; Liu, S.; Wang, Y.; and Fardad, M. 2019. Generation of low distortion adversarial attacks via convex programming. In 2019 IEEE International Conference on Data Mining (ICDM), 1486--1491. IEEE
2019
-
[36]
Zhang, Z.; Yu, W.; Yu, M.; Guo, Z.; and Jiang, M. 2022. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. arXiv preprint arXiv:2204.03508
2022 arXiv
-
[37]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[38]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.