Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read ABBG is a white-box attack that uses only the tracker's predicted bounding box and drives TransT-M, ROMTrack, and MixFormer to near-zero overlap on GOT-10k.

desk verdict The single-bbox proxy is genuinely novel and the results are striking, but the paper under-specifies the loss graph; the zero-gradient critique is overstated because scale gradients alone can shrink the box to near-zero overlap. read the letter →

arxiv 2411.17468 v1 pith:6DYNLKFH submitted 2024-11-26 cs.CV

classification cs.CV
keywords adversarialattackvisualobjecttrackingtransformertrackerswhite-boxboundingboxregressionGOT-10krobustnessIntersectionoverUnion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a simple fact: to attack a transformer-based visual object tracker in a white-box setting, the tracker's single predicted bounding box is enough, with no classification labels, regression labels, heatmaps, or masks required. From that one box, the method samples thousands of random adversarial boxes, keeps those with high IoU with the prediction, and backpropagates a smoothed-L1 regression loss through the tracker to perturb the search region. On GOT-10k the attack reduces TransT-M from 0.734 to 0.027 average overlap, ROMTrack from 0.729 to 0.006, and MixFormer from 0.696 to 0.002, while SPARK, RTAA, and TrackPGD cannot even be applied to ROMTrack and MixFormer. The paper concludes that transformer trackers can be seriously challenged by an attacker that uses nothing beyond the bounding box prediction, opening these trackers to standardized white-box robustness comparisons.

What carries the argument

The load-bearing object is the set of Adversarial Bounding Boxes (ABBs) generated from the predicted box. Each step samples $k=1024$ boxes $b_i^*$ with $x_i' = x + T_{x_i}$, $y_i'=y+T_{y_i}$, $w_i'=w s_i$, $h_i'=h s_i$ from uniform distributions, then keeps only boxes with IoU above an adaptive threshold that always retains about 80% of the samples so the loss is never empty. The attack loss is $\ell_{\mathrm{ABBG}} = \sum_i \ell_r(b_{\mathrm{pred}}, b_i^*)$ with $\ell_r$ the smoothed L1 norm; this regression loss is backpropagated through the transformer to produce the perturbation, iterated 10 times with $\epsilon=10$ clipping. The random sampling plus thresholding is what turns a single non-probabilistic box output into a differentiable training signal.

What would settle it

Take a tracker with the same weights and replace only the final box decoding with a non-differentiable operation such as rounding the box coordinates or using argmax over a fixed grid, then run ABBG with the same budget. If the average overlap on GOT-10k stays near the clean values instead of collapsing to the reported 0.027, 0.006, and 0.002, the claim that one bounding box is sufficient as a white-box proxy is refuted. A second check: apply a random perturbation of the same $\epsilon=10$ budget; if it alone collapses the scores, the specific regression loss is not the cause.

Watch

Extended reading notes

Core claim

The central claim is that a transformer tracker's own predicted bounding box is a sufficient white-box attack proxy. ABBG samples 1,024 adversarial boxes around the prediction using random translations and scales, keeps the subset whose Intersection over Union with the predicted box lies above an adaptive threshold so that about 80% of the sampled boxes remain positive samples, and minimizes the smoothed-L1 regression loss between the prediction and those retained boxes by backpropagating through the tracker. In a white-box setting this drives the three tested trackers to near failure: on GOT-10k, TransT-M average overlap falls from 0.734 to 0.027, ROMTrack from 0.729 to 0.006, and MixFormer from 0.696 to 0.002. The same attack also applies to TransT-M on UAV123 and VOT2022-ST, and unlike SPARK, RTAA, and TrackPGD, it does not require classification or regression labels, heatmaps, or binary masks, which are absent from ROMTrack and MixFormer pipelines.

Load-bearing premise

The attack only works if the tracker's predicted bounding box is a smooth, learnable function of the input image; if the box comes from any step that blocks the gradient, or if the sampled boxes are recomputed so the loss stops changing, the perturbation has no gradient to follow and collapses.

Editorial extensions

If this is right

  • ABBG gives a single white-box attack that runs on TransT-M, ROMTrack, and MixFormer, so the adversarial robustness of these transformer trackers can be compared on the same footing.
  • On ROMTrack and MixFormer, the attack drives SR0.5 and SR0.75 to zero, meaning the tracker fails even coarse overlap, not just precise localization.
  • Because the only proxy is the predicted box, any transformer tracker that returns a differentiable box prediction is a candidate target, no matter what other outputs its pipeline does or does not provide.
  • The attack achieves these drops within the same 10-iteration, epsilon=10 budget as SPARK and RTAA, while ranking second in sparsity and imperceptibility among white-box attacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-box regression loss is the actual driver of the collapse, then defenses should harden the box-regression head and the gradient path, not the classification or score branches; this is a direction the paper does not itself propose.
  • The adaptive threshold always keeping roughly 80% of sampled boxes suggests the attack is a form of smoothed gradient over the box-output manifold; a natural testable variant would sample boxes from a deterministic grid to see whether randomness or thresholding matters more.
  • The same 'perturb the tracker's own output' recipe could transfer to other differentiable output heads, such as keypoints, masks, or rotated boxes, where no candidate list exists, although the paper does not test this.
  • If a tracker detached or non-differentiably decoded its box, ABBG would likely fail, which points to a cheap but possibly harmful defense: cutting gradients from the box head.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ABBG, a white-box adversarial attack against transformer-based visual object trackers, using only the tracker's predicted bounding box as the attack proxy. The method generates k random translated/scaled copies of the predicted box, computes a smoothed L1 regression loss between the predicted box and these adversarial boxes, and backpropagates the loss through the tracker to perturb the input search region. Experiments on TransT-M, ROMTrack, and MixFormer over GOT-10k, UAV123, and VOT2022-ST report large performance drops, e.g., TransT-M AO from 0.734 to 0.027 on GOT-10k, with the method claimed to be applicable to trackers where SPARK, RTAA, and TrackPGD cannot be used.

Significance. If valid, the work would fill a real gap: a white-box attack that works with only a single bounding box output would enable robustness evaluation of modern transformer trackers that lack classification/regression heads or binary masks. The paper also attempts a comparison with existing white-box attacks and reports sparsity/imperceptibility metrics. However, the central attack formulation is under-specified, and as written the loss cannot generate the gradients needed to move the predicted box, which directly undermines the reported results. No code or detailed computation graph is provided, so the experiments are not reproducible from the text. The contribution is therefore not established.

major comments (3)
  1. [Section 3, Eq. (1) and Eq. (2)] The loss ℓABBG = Σ_i ℓr(bpred, b_i*) is computed with b_i* generated from bpred via Eq. (1): x'_i = x + Tx_i, y'_i = y + Ty_i, w'_i = w s_i, h'_i = h s_i. For the translation coordinates, ℓr(x, x+Tx_i) and ℓr(y, y+Ty_i) are independent of x and y, so ∂ℓABBG/∂x = ∂ℓABBG/∂y = 0. The width and height terms retain a gradient through w(1−s_i) and h(1−s_i), but that can only shrink or grow the box, not shift it. The reported drops to near-zero AO in Table 1 require the predicted box to move far from the target. Either the paper must specify that b_i* is generated from a detached prediction (or from a fixed clean anchor), or it must provide a computation graph showing how the translation gradients arise. As written, the attack cannot work, and the experimental numbers are unexplained.
  2. [Section 3, Figure 2] The adaptive IoU-based selection of positive and negative samples is not defined precisely enough to be implemented or differentiated. The text says 'an adaptive threshold that only retains bounding boxes with an Intersection over Union (IoU) greater than a fixed threshold, specifically 80% of all adversarial bounding boxes,' which is self-contradictory: the threshold is both adaptive and fixed. It is also unclear whether the selected set is a hard mask and how the non-differentiable selection is treated during backpropagation.
  3. [Section 4.1, Tables 1 and 2] The central claim that ABBG is 'the first white-box attack that is applicable to a wide range of trackers with transformer backbones' is not substantiated by the experiments. For ROMTrack and MixFormer, the tables report only CSA, IoU, and ABBG; SPARK, RTAA, and TrackPGD are evaluated only on TransT-M. To support the applicability claim, the authors should either run the other white-box attacks on ROMTrack and MixFormer or provide concrete, code-level evidence of why those attacks cannot be applied.
minor comments (5)
  1. [Abstract and throughout] The dataset name 'UA V123' should be 'UAV123' (the typo also appears in the abstract and in Tables 1 and 3).
  2. [Section 2] There is a typo in 'The use of vision transformers in the backbone of tackers' — 'tackers' should be 'trackers'.
  3. [Section 3] The phrase 'closest to the the predicted bounding box' contains a duplicated 'the'.
  4. [Section 4, Attack setup] The units of the translation parameters (sampled uniformly between 0.1 and 0.4) are not specified. They need to be defined as absolute pixels or as fractions of the box width/height; otherwise, the setup is not reproducible.
  5. [Section 4, Attack setup] The paper states that the number of iterations is 10 and the perturbation is clipped to an ϵ-ball with ϵ = 10, but it does not specify the step size or the exact PGD update rule, which are needed for reproduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the ABBG attack's benchmark drops are externally evaluated, and its loss is not equivalent to its inputs by construction.

full rationale

I walked the claimed derivation chain. The attack defines its loss as ℓABBG = Σ ℓr(bpred, b_i*), where the adversarial boxes b_i* are generated from the tracker's own predicted box via Eq. 1. This is the intended mechanism of a white-box attack, not a derivation of a predictive claim from its inputs. The paper's central claims—near-zero AO and large drops on GOT-10k, UAV123, and VOT2022-ST—are measured against external ground-truth benchmarks, so they are empirical outcomes rather than quantities forced by fitted constants. The hyperparameters (k=1024, translation ranges 0.1–0.4, scale range 0.7–0.9, 10 iterations) are hand-selected and are not fitted to the reported metrics. The self-citations (TrackPGD and the reproducibility study) motivate the gap and provide a comparison baseline, but the ABBG loss is independently defined in Eqs. 1–2 and does not reduce to those prior works. A technical nuance is that the translation components of the loss have zero gradient with respect to the predicted box's x,y coordinates, leaving the scale terms as the operative gradient signal; this affects attack mechanics and reproducibility, but it is not circularity, because the benchmark results are externally evaluated and the scale-only gradient can plausibly shrink the predicted box to produce the reported overlap drops. No step in the paper reduces a prediction to its own inputs, and no load-bearing self-citation chain is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or model entities. The central claim rests on a set of hand-selected attack hyperparameters and on the assumption that a single bounding box output is a differentiable attack proxy. The loss equation as written is not enough to guarantee a nonzero gradient, which is the main unresolved assumption.

free parameters (4)
  • k (number of adversarial boxes) = 1024
    Chosen by hand; controls the number of random boxes and the gradient signal; no ablation reported (Section 4, Attack setup).
  • Translation range (Tx, Ty) = U(0.1, 0.4)
    Hand-selected range for random box shifts; attack effectiveness depends on it; no sensitivity analysis provided.
  • Scale range (S) = U(0.7, 0.9)
    Hand-selected range for random box scaling; not justified by data or theory.
  • IoU retention threshold = 80% of boxes with highest IoU
    Adaptive threshold for selecting positive samples; the exact rule is under-specified in Section 3.
assumptions (4)
  • domain assumption Tracker bounding box output is differentiable with respect to the input search region through the whole transformer.
    Required for white-box gradient backpropagation in Section 3 and Figure 2; not verified for non-differentiable decoding steps.
  • domain assumption The selected transformer trackers (TransT-M, ROMTrack, MixFormer) provide access to a single predicted bounding box suitable as an attack proxy.
    The method's applicability claim rests on this interface (Section 1).
  • domain assumption The official benchmark evaluation protocols (GOT-10k, VOT2022-STS, UAV123) are correctly implemented and clean scores match published baselines.
    Attack drop percentages are computed relative to 'No Attack' rows in Tables 1-3; no verification of clean scores against official numbers is reported.
  • domain assumption A single stochastic run is representative of attack performance.
    The attack samples random boxes each step, yet results are reported without variance, seeds, or repeated runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers." pith.science (2026). https://pith.science/paper/6DYNLKFH

@misc{pith2026241117468,
  author       = {Pith},
  title        = {Pith review of: Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DYNLKFH}},
  note         = {Machine review of arXiv:2411.17468}
}
read the original abstract

Adversarial perturbations aim to deceive neural networks into predicting inaccurate results. For visual object trackers, adversarial attacks have been developed to generate perturbations by manipulating the outputs. However, transformer trackers predict a specific bounding box instead of an object candidate list, which limits the applicability of many existing attack scenarios. To address this issue, we present a novel white-box approach to attack visual object trackers with transformer backbones using only one bounding box. From the tracker predicted bounding box, we generate a list of adversarial bounding boxes and compute the adversarial loss for those bounding boxes. Experimental results demonstrate that our simple yet effective attack outperforms existing attacks against several robust transformer trackers, including TransT-M, ROMTrack, and MixFormer, on popular benchmark tracking datasets such as GOT-10k, UAV123, and VOT2022STS.

Figures

Figures reproduced from arXiv: 2411.17468 by the authors.

Figure 1
Figure 1. The adversarial robustness of transformer-based trackers, including ROMTrack ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of the ABBG attack approach. The random sets of scale [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Several examples of TransT-M (Chen et al., 2023) performance after applying the white-box attacks containing SPARK (Guo et al., 2020) (blue), RTAA (Jia et al., 2020) (black), TrackPGD (Nok￾abadi et al., 2024) (white), and our proposed ABBG attack (red) bounding boxes. The Green color represents the tracker’s original response with no attack applied [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Several examples of TransT-M (Chen et al., 2023) performance after applying the white-box attacks containing SPARK (Guo et al., 2020) (blue), RTAA (Jia et al., 2020) (black), TrackPGD (Nok￾abadi et al., 2024) (white), and our proposed ABBG attack (red) bounding boxes. …
Figure 5
Figure 5. Figure 5: Several examples of TransT-M (Chen et al., 2023) performance after applying the white-box attacks containing SPARK (Guo et al., 2020) (blue), RTAA (Jia et al., 2020) (black), TrackPGD (Nok￾abadi et al., 2024) (white), and our proposed ABBG attack (red) binary masks. Th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

  1. [1]

    The Tenth Visual Object Tracking VOT2022 Challenge Results

    Kristan, M.; Leonardis, A.; Matas, J.; Felsberg, M. The Tenth Visual Object Tracking VOT2022 Challenge Results. IEEE Conf. Euro. Conf. Comput. Vis. Worksh. 2023

  2. [2]

    MixFormer: End-to-End Tracking With Iterative Mixed Attention

    Cui, Y.; Jiang, C.; Wang, L.; Wu, G. MixFormer: End-to-End Tracking With Iterative Mixed Attention. IEEE Conf. on Comput. Vis. and Pattern Recog. 2022

  3. [3]

    High-Performance Transformer Tracking

    Chen, X.; Yan, B.; Zhu, J.; Lu, H.; Ruan, X.; Wang, D. High-Performance Transformer Tracking. IEEE Trans. on Pattern Analy. and Machine Intel. 2023, 45, 8507--8523

  4. [4]

    Robust Object Modeling for Visual Tracking

    Cai, Y.; Liu, J.; Tang, J.; Wu, G. Robust Object Modeling for Visual Tracking. IEEE Conf. on Comput. Vis. 2023

  5. [5]

    N.; Lalonde, J.-F.; Gagn \'e , C

    Nokabadi, F. N.; Lalonde, J.-F.; Gagn \'e , C. Reproducibility Study on Adversarial Attacks Against Robust Transformer Trackers. Trans. on Mach. Learn. Research. 2024

  6. [6]

    SPARK: Spatial-Aware Online Incremental Attack Against Visual Tracking

    Guo, Q.; Xie, X.; Juefei-Xu, F.; Ma, L.; Li, Z.; Xue, W.; Feng, W.; Liu, Y. SPARK: Spatial-Aware Online Incremental Attack Against Visual Tracking. IEEE Conf. Euro. Conf. Comput. Vis. 2020

  7. [7]

    Robust Tracking Against Adversarial Attacks

    Jia, S.; Ma, C.; Song, Y.; Yang, X. Robust Tracking Against Adversarial Attacks. IEEE Conf. Euro. Conf. Comput. Vis. 2020

  8. [8]

    Cooling-Shrinking Attack: Blinding the Tracker With Imperceptible Noises

    Yan, B.; Wang, D.; Lu, H.; Yang, X. Cooling-Shrinking Attack: Blinding the Tracker With Imperceptible Noises. IEEE Conf. on Comput. Vis. and Pattern Recog. 2020

Show all 16 references
  1. [9]

    Transformer Tracking

    Chen, X.; Yan, B.; Zhu, J.; Wang, D.; Yang, X.; Lu, H. Transformer Tracking. IEEE Conf. on Comput. Vis. and Pattern Recog. 2021

  2. [10]

    N.; Yann Batiste, P.; Lalonde, J.-F.; Gagn \'e , C

    Nokabadi, F. N.; Yann Batiste, P.; Lalonde, J.-F.; Gagn \'e , C. TrackPGD: A White-box Attack using Binary Masks against Robust Transformer Trackers. arXiv preprint arXiv:2407.03946 2024,

  3. [11]

    GOT -10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild

    Huang, L.; Zhao, X.; Huang, K. GOT -10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild. IEEE Trans. on Pattern Analy. and Machine Intel. 2019, 43, 1562--1577

  4. [12]

    Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework

    Ye, B.; Chang, H.; Ma, B.; Shan, S.; Chen, X. Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework. IEEE Conf. Euro. Conf. Comput. Vis. 2022

  5. [13]

    IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking

    Jia, S.; Song, Y.; Ma, C.; Yang, X. IoU Attack: Towards Temporally Coherent Black-Box Adversarial Attack for Visual Object Tracking. IEEE Conf. on Comput. Vis. and Pattern Recog. 2021

  6. [14]

    A Benchmark and Simulator for UAV Tracking

    Mueller, M.; Smith, N.; Ghanem, B. A Benchmark and Simulator for UAV Tracking. IEEE Conf. Euro. Conf. Comput. Vis. 2016

  7. [15]

    Image quality assessment: from error visibility to structural similarity

    Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE Trans. on Image Process. 2004, 13, 600--612

  8. [16]

    tracking failure

    Li, B.; Wu, W.; Wang, Q.; Zhang, F.; Xing, J.; Yan, J. SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. IEEE Conf. on Comput. Vis. and Pattern Recog. 2019 mcitethebibliography main.tex0000664000000000000000000002242514721355247011243 0ustar rootroot art...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.