REVIEW 3 major objections 5 minor 16 references
Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read ABBG is a white-box attack that uses only the tracker's predicted bounding box and drives TransT-M, ROMTrack, and MixFormer to near-zero overlap on GOT-10k.
desk verdict The single-bbox proxy is genuinely novel and the results are striking, but the paper under-specifies the loss graph; the zero-gradient critique is overstated because scale gradients alone can shrink the box to near-zero overlap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the set of Adversarial Bounding Boxes (ABBs) generated from the predicted box. Each step samples $k=1024$ boxes $b_i^*$ with $x_i' = x + T_{x_i}$, $y_i'=y+T_{y_i}$, $w_i'=w s_i$, $h_i'=h s_i$ from uniform distributions, then keeps only boxes with IoU above an adaptive threshold that always retains about 80% of the samples so the loss is never empty. The attack loss is $\ell_{\mathrm{ABBG}} = \sum_i \ell_r(b_{\mathrm{pred}}, b_i^*)$ with $\ell_r$ the smoothed L1 norm; this regression loss is backpropagated through the transformer to produce the perturbation, iterated 10 times with $\epsilon=10$ clipping. The random sampling plus thresholding is what turns a single non-probabilistic box output into a differentiable training signal.
What would settle it
Take a tracker with the same weights and replace only the final box decoding with a non-differentiable operation such as rounding the box coordinates or using argmax over a fixed grid, then run ABBG with the same budget. If the average overlap on GOT-10k stays near the clean values instead of collapsing to the reported 0.027, 0.006, and 0.002, the claim that one bounding box is sufficient as a white-box proxy is refuted. A second check: apply a random perturbation of the same $\epsilon=10$ budget; if it alone collapses the scores, the specific regression loss is not the cause.
Extended reading notes
Core claim
The central claim is that a transformer tracker's own predicted bounding box is a sufficient white-box attack proxy. ABBG samples 1,024 adversarial boxes around the prediction using random translations and scales, keeps the subset whose Intersection over Union with the predicted box lies above an adaptive threshold so that about 80% of the sampled boxes remain positive samples, and minimizes the smoothed-L1 regression loss between the prediction and those retained boxes by backpropagating through the tracker. In a white-box setting this drives the three tested trackers to near failure: on GOT-10k, TransT-M average overlap falls from 0.734 to 0.027, ROMTrack from 0.729 to 0.006, and MixFormer from 0.696 to 0.002. The same attack also applies to TransT-M on UAV123 and VOT2022-ST, and unlike SPARK, RTAA, and TrackPGD, it does not require classification or regression labels, heatmaps, or binary masks, which are absent from ROMTrack and MixFormer pipelines.
Load-bearing premise
The attack only works if the tracker's predicted bounding box is a smooth, learnable function of the input image; if the box comes from any step that blocks the gradient, or if the sampled boxes are recomputed so the loss stops changing, the perturbation has no gradient to follow and collapses.
Editorial extensions
If this is right
- ABBG gives a single white-box attack that runs on TransT-M, ROMTrack, and MixFormer, so the adversarial robustness of these transformer trackers can be compared on the same footing.
- On ROMTrack and MixFormer, the attack drives SR0.5 and SR0.75 to zero, meaning the tracker fails even coarse overlap, not just precise localization.
- Because the only proxy is the predicted box, any transformer tracker that returns a differentiable box prediction is a candidate target, no matter what other outputs its pipeline does or does not provide.
- The attack achieves these drops within the same 10-iteration, epsilon=10 budget as SPARK and RTAA, while ranking second in sparsity and imperceptibility among white-box attacks.
Reading between the lines
- If the single-box regression loss is the actual driver of the collapse, then defenses should harden the box-regression head and the gradient path, not the classification or score branches; this is a direction the paper does not itself propose.
- The adaptive threshold always keeping roughly 80% of sampled boxes suggests the attack is a form of smoothed gradient over the box-output manifold; a natural testable variant would sample boxes from a deterministic grid to see whether randomness or thresholding matters more.
- The same 'perturb the tracker's own output' recipe could transfer to other differentiable output heads, such as keypoints, masks, or rotated boxes, where no candidate list exists, although the paper does not test this.
- If a tracker detached or non-differentiably decoded its box, ABBG would likely fail, which points to a cheap but possibly harmful defense: cutting gradients from the box head.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ABBG, a white-box adversarial attack against transformer-based visual object trackers, using only the tracker's predicted bounding box as the attack proxy. The method generates k random translated/scaled copies of the predicted box, computes a smoothed L1 regression loss between the predicted box and these adversarial boxes, and backpropagates the loss through the tracker to perturb the input search region. Experiments on TransT-M, ROMTrack, and MixFormer over GOT-10k, UAV123, and VOT2022-ST report large performance drops, e.g., TransT-M AO from 0.734 to 0.027 on GOT-10k, with the method claimed to be applicable to trackers where SPARK, RTAA, and TrackPGD cannot be used.
Significance. If valid, the work would fill a real gap: a white-box attack that works with only a single bounding box output would enable robustness evaluation of modern transformer trackers that lack classification/regression heads or binary masks. The paper also attempts a comparison with existing white-box attacks and reports sparsity/imperceptibility metrics. However, the central attack formulation is under-specified, and as written the loss cannot generate the gradients needed to move the predicted box, which directly undermines the reported results. No code or detailed computation graph is provided, so the experiments are not reproducible from the text. The contribution is therefore not established.
major comments (3)
- [Section 3, Eq. (1) and Eq. (2)] The loss ℓABBG = Σ_i ℓr(bpred, b_i*) is computed with b_i* generated from bpred via Eq. (1): x'_i = x + Tx_i, y'_i = y + Ty_i, w'_i = w s_i, h'_i = h s_i. For the translation coordinates, ℓr(x, x+Tx_i) and ℓr(y, y+Ty_i) are independent of x and y, so ∂ℓABBG/∂x = ∂ℓABBG/∂y = 0. The width and height terms retain a gradient through w(1−s_i) and h(1−s_i), but that can only shrink or grow the box, not shift it. The reported drops to near-zero AO in Table 1 require the predicted box to move far from the target. Either the paper must specify that b_i* is generated from a detached prediction (or from a fixed clean anchor), or it must provide a computation graph showing how the translation gradients arise. As written, the attack cannot work, and the experimental numbers are unexplained.
- [Section 3, Figure 2] The adaptive IoU-based selection of positive and negative samples is not defined precisely enough to be implemented or differentiated. The text says 'an adaptive threshold that only retains bounding boxes with an Intersection over Union (IoU) greater than a fixed threshold, specifically 80% of all adversarial bounding boxes,' which is self-contradictory: the threshold is both adaptive and fixed. It is also unclear whether the selected set is a hard mask and how the non-differentiable selection is treated during backpropagation.
- [Section 4.1, Tables 1 and 2] The central claim that ABBG is 'the first white-box attack that is applicable to a wide range of trackers with transformer backbones' is not substantiated by the experiments. For ROMTrack and MixFormer, the tables report only CSA, IoU, and ABBG; SPARK, RTAA, and TrackPGD are evaluated only on TransT-M. To support the applicability claim, the authors should either run the other white-box attacks on ROMTrack and MixFormer or provide concrete, code-level evidence of why those attacks cannot be applied.
minor comments (5)
- [Abstract and throughout] The dataset name 'UA V123' should be 'UAV123' (the typo also appears in the abstract and in Tables 1 and 3).
- [Section 2] There is a typo in 'The use of vision transformers in the backbone of tackers' — 'tackers' should be 'trackers'.
- [Section 3] The phrase 'closest to the the predicted bounding box' contains a duplicated 'the'.
- [Section 4, Attack setup] The units of the translation parameters (sampled uniformly between 0.1 and 0.4) are not specified. They need to be defined as absolute pixels or as fractions of the box width/height; otherwise, the setup is not reproducible.
- [Section 4, Attack setup] The paper states that the number of iterations is 10 and the perturbation is clipped to an ϵ-ball with ϵ = 10, but it does not specify the step size or the exact PGD update rule, which are needed for reproduction.
Circularity Check
No circularity: the ABBG attack's benchmark drops are externally evaluated, and its loss is not equivalent to its inputs by construction.
full rationale
I walked the claimed derivation chain. The attack defines its loss as ℓABBG = Σ ℓr(bpred, b_i*), where the adversarial boxes b_i* are generated from the tracker's own predicted box via Eq. 1. This is the intended mechanism of a white-box attack, not a derivation of a predictive claim from its inputs. The paper's central claims—near-zero AO and large drops on GOT-10k, UAV123, and VOT2022-ST—are measured against external ground-truth benchmarks, so they are empirical outcomes rather than quantities forced by fitted constants. The hyperparameters (k=1024, translation ranges 0.1–0.4, scale range 0.7–0.9, 10 iterations) are hand-selected and are not fitted to the reported metrics. The self-citations (TrackPGD and the reproducibility study) motivate the gap and provide a comparison baseline, but the ABBG loss is independently defined in Eqs. 1–2 and does not reduce to those prior works. A technical nuance is that the translation components of the loss have zero gradient with respect to the predicted box's x,y coordinates, leaving the scale terms as the operative gradient signal; this affects attack mechanics and reproducibility, but it is not circularity, because the benchmark results are externally evaluated and the scale-only gradient can plausibly shrink the predicted box to produce the reported overlap drops. No step in the paper reduces a prediction to its own inputs, and no load-bearing self-citation chain is present.
Assumptions & free parameters
free parameters (4)
- k (number of adversarial boxes) =
1024
- Translation range (Tx, Ty) =
U(0.1, 0.4)
- Scale range (S) =
U(0.7, 0.9)
- IoU retention threshold =
80% of boxes with highest IoU
assumptions (4)
- domain assumption Tracker bounding box output is differentiable with respect to the input search region through the whole transformer.
- domain assumption The selected transformer trackers (TransT-M, ROMTrack, MixFormer) provide access to a single predicted bounding box suitable as an attack proxy.
- domain assumption The official benchmark evaluation protocols (GOT-10k, VOT2022-STS, UAV123) are correctly implemented and clean scores match published baselines.
- domain assumption A single stochastic run is representative of attack performance.
Cite this review
Pith. "Pith review of Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers." pith.science (2026). https://pith.science/paper/6DYNLKFH
@misc{pith2026241117468,
author = {Pith},
title = {Pith review of: Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers},
year = {2026},
howpublished = {\url{https://pith.science/paper/6DYNLKFH}},
note = {Machine review of arXiv:2411.17468}
}
read the original abstract
Adversarial perturbations aim to deceive neural networks into predicting inaccurate results. For visual object trackers, adversarial attacks have been developed to generate perturbations by manipulating the outputs. However, transformer trackers predict a specific bounding box instead of an object candidate list, which limits the applicability of many existing attack scenarios. To address this issue, we present a novel white-box approach to attack visual object trackers with transformer backbones using only one bounding box. From the tracker predicted bounding box, we generate a list of adversarial bounding boxes and compute the adversarial loss for those bounding boxes. Experimental results demonstrate that our simple yet effective attack outperforms existing attacks against several robust transformer trackers, including TransT-M, ROMTrack, and MixFormer, on popular benchmark tracking datasets such as GOT-10k, UAV123, and VOT2022STS.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
The Tenth Visual Object Tracking VOT2022 Challenge Results
Kristan, M.; Leonardis, A.; Matas, J.; Felsberg, M. The Tenth Visual Object Tracking VOT2022 Challenge Results. IEEE Conf. Euro. Conf. Comput. Vis. Worksh. 2023
work page 2023
-
[2]
MixFormer: End-to-End Tracking With Iterative Mixed Attention
Cui, Y.; Jiang, C.; Wang, L.; Wu, G. MixFormer: End-to-End Tracking With Iterative Mixed Attention. IEEE Conf. on Comput. Vis. and Pattern Recog. 2022
work page 2022
-
[3]
High-Performance Transformer Tracking
Chen, X.; Yan, B.; Zhu, J.; Lu, H.; Ruan, X.; Wang, D. High-Performance Transformer Tracking. IEEE Trans. on Pattern Analy. and Machine Intel. 2023, 45, 8507--8523
work page 2023
-
[4]
Robust Object Modeling for Visual Tracking
Cai, Y.; Liu, J.; Tang, J.; Wu, G. Robust Object Modeling for Visual Tracking. IEEE Conf. on Comput. Vis. 2023
work page 2023
-
[5]
N.; Lalonde, J.-F.; Gagn \'e , C
Nokabadi, F. N.; Lalonde, J.-F.; Gagn \'e , C. Reproducibility Study on Adversarial Attacks Against Robust Transformer Trackers. Trans. on Mach. Learn. Research. 2024
work page 2024
-
[6]
SPARK: Spatial-Aware Online Incremental Attack Against Visual Tracking
Guo, Q.; Xie, X.; Juefei-Xu, F.; Ma, L.; Li, Z.; Xue, W.; Feng, W.; Liu, Y. SPARK: Spatial-Aware Online Incremental Attack Against Visual Tracking. IEEE Conf. Euro. Conf. Comput. Vis. 2020
work page 2020
-
[7]
Robust Tracking Against Adversarial Attacks
Jia, S.; Ma, C.; Song, Y.; Yang, X. Robust Tracking Against Adversarial Attacks. IEEE Conf. Euro. Conf. Comput. Vis. 2020
work page 2020
-
[8]
Cooling-Shrinking Attack: Blinding the Tracker With Imperceptible Noises
Yan, B.; Wang, D.; Lu, H.; Yang, X. Cooling-Shrinking Attack: Blinding the Tracker With Imperceptible Noises. IEEE Conf. on Comput. Vis. and Pattern Recog. 2020
work page 2020
Show all 16 references
-
[9]
Transformer Tracking
Chen, X.; Yan, B.; Zhu, J.; Wang, D.; Yang, X.; Lu, H. Transformer Tracking. IEEE Conf. on Comput. Vis. and Pattern Recog. 2021
2021
-
[10]
N.; Yann Batiste, P.; Lalonde, J.-F.; Gagn \'e , C
Nokabadi, F. N.; Yann Batiste, P.; Lalonde, J.-F.; Gagn \'e , C. TrackPGD: A White-box Attack using Binary Masks against Robust Transformer Trackers. arXiv preprint arXiv:2407.03946 2024,
2024 arXiv
-
[11]
GOT -10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
Huang, L.; Zhao, X.; Huang, K. GOT -10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild. IEEE Trans. on Pattern Analy. and Machine Intel. 2019, 43, 1562--1577
2019
-
[12]
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework
Ye, B.; Chang, H.; Ma, B.; Shan, S.; Chen, X. Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework. IEEE Conf. Euro. Conf. Comput. Vis. 2022
2022
-
[13]
IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking
Jia, S.; Song, Y.; Ma, C.; Yang, X. IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking. IEEE Conf. on Comput. Vis. and Pattern Recog. 2021
2021
-
[14]
A Benchmark and Simulator for UAV Tracking
Mueller, M.; Smith, N.; Ghanem, B. A Benchmark and Simulator for UAV Tracking. IEEE Conf. Euro. Conf. Comput. Vis. 2016
2016
-
[15]
Image quality assessment: from error visibility to structural similarity
Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE Trans. on Image Process. 2004, 13, 600--612
2004
-
[16]
tracking failure
Li, B.; Wu, W.; Wang, Q.; Zhang, F.; Xing, J.; Yan, J. SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. IEEE Conf. on Comput. Vis. and Pattern Recog. 2019 mcitethebibliography main.tex0000664000000000000000000002242514721355247011243 0ustar rootroot art...
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.