Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Model Decay in Long-Term Tracking

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Updating a tracker on its own predictions adds an unavoidable bias—model decay—that grows linearly with past errors, and this accumulated bias is what makes long-term trackers drift and lose the target.

desk verdict The empirical long-term tracker is worth a look, but the central derivation of 'model decay' rests on a wrong linearization, so the theory should not be taken as proven. read the letter →

arxiv 1908.01603 v1 pith:J463SWKL submitted 2019-08-05 cs.CV

classification cs.CV
keywords modeldecaylong-termtrackingvisualobjectonlinelearningbiasSiamesetrackerscorrelationfiltersself-trainingdriftbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual object trackers that keep updating their model using their own bounding-box predictions as training labels are adding an unavoidable bias to the learning, a bias the authors call model decay. The paper derives this bias from the standard gradient-update equations and shows that the model output shifts by a quantity linearly proportional to past prediction errors. Because the bias accumulates recursively over time, short tracking videos hide it while long videos expose it as drift and eventual target loss. The authors show empirically that a Siamese tracker whose updates are gated by a separately trained 'decay recognition network' reduces this decay and stays accurate on videos up to 30 minutes long. If the derivation is right, long-term tracking failures are not a bug in any single tracker but a mathematical property of self-training.

What carries the argument

The key machinery is the linearized model-dynamics identity that decomposes the change in a tracker's model output after an online gradient step into a perfect-update term and a model-decay term $2\eta\mathbb{E}[\delta_{i,t}\|\nabla_\varphi f_{i,t}\|^2]$, which is linear in the past prediction errors $\delta_{i,t}$ and weighted by the squared sensitivity of the model output to its parameters. This identity turns the vague intuition that 'bad updates cause drift' into a concrete, recursive quantity that accumulates over time and can be measured. The paper's practical counterpart is the 'decay recognition network', an LSTM trained on Siamese similarity maps that decides whether a candidate update is likely to add bias, thereby approximating a gate on whether the model should be updated at all.

What would settle it

Run an online-updating tracker on a long video with known ground truth, and at every update record the prediction error $\delta_i$, the model-output gradient $\nabla_\varphi f_{i,t}$, and the actual change in the model's predicted box $f_{i,t+1}-f_{i,t}$. The theory predicts that the component of this change beyond the 'perfect update' equals $2\eta\,\mathbb{E}[\delta_{i,t}\|\nabla_\varphi f_{i,t}\|^2]$. If the measured shift does not match this linear, gradient-squared-weighted form—for instance, if it depends on other quantities or does not scale with $2\eta$—then the model-decay term as derived is not the mechanism driving drift.

Watch

Extended reading notes

Core claim

The central discovery is that the online update of any tracker model trained on its own outputs splits into two components: a perfect update that corrects the model toward the true target box, and a bias term proportional to the noise in past predictions. Concretely, after one gradient step the model output at a frame $i$ changes as $f_{i,t+1}=f_{i,t}-2\eta\mathbb{E}[(f_{i,t}-y^*_i)\|\nabla_\varphi f_{i,t}\|^2]+2\eta\mathbb{E}[\delta_{i,t}\|\nabla_\varphi f_{i,t}\|^2]$, where the first term is the 'perfect model update' and the second term, the 'model decay', offshoots the model by a quantity linearly proportional to past errors $\delta_i$. Because this decay term accumulates recursively, any tracker that keeps updating on imperfect self-predictions drifts, and the drift is a mathematical consequence of self-training rather than a failure of a particular algorithm. The authors demonstrate the effect on correlation-filter and Siamese trackers, and use the insight to build LT-SINT, which gates updates with a learned decay-recognition network and global search, reporting that it avoids significant decay even in 30-minute videos.

Load-bearing premise

The central derivation assumes that the way a tracker's output changes when its parameters change can be summarized by the squared size of that change; if that algebraic step is not valid for a given model, the claimed linear growth of model decay with past errors does not follow.

Editorial extensions

If this is right

  • Any tracker that re-trains on its own predictions carries a built-in drift: the model-decay term is unavoidable under gradient updates, regardless of architecture.
  • The decay grows with the number of updates and the magnitude of past errors, which is why 10-to-20-second benchmarks hide the problem while 30-minute videos expose it as target loss.
  • A tracker that never updates, such as a pure Siamese matcher, has zero model decay but cannot adapt to appearance change; LT-SINT shows sparse, gated updates as a working middle ground.
  • Gating updates with a separately trained decay-recognition network improves AUC on UAV20L, YouTubeLong, and OxUvA compared with both updating every frame and never updating.
  • The approach also scores 59.8 AUC on the short-term OTB benchmark, close to popular short-term trackers, indicating long-term robustness does not sacrifice short-term utility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same bias term should appear in any online-learning system that trains on its own outputs, such as self-training language models or bootstrapped reinforcement learning, so the linear-growth formula gives a concrete way to test for 'model collapse' or policy drift in those settings.
  • Beyond the paper: because the decay term is weighted by $\|\nabla_\varphi f_{i,t}\|^2$, reducing a model's output sensitivity—via regularization, smaller learning rates, or flatter minima—should slow decay even without a gating network; the paper does not test this prediction.
  • Beyond the paper: the decay-recognition network is a general pattern—an external meta-controller that decides when a learner should update, trained on the learner's confidence signals rather than its parameters—that could transfer to continual learning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the phenomenon of model decay in long-term visual object tracking. It presents a mathematical analysis of the learning dynamics of tracker models, claiming that any tracker updating on its own predictions accumulates a bias term—model decay—that is linearly proportional to past prediction errors. Based on this analysis, the authors propose LT-SINT, a Siamese tracker augmented with a decay recognition network (DRN) and a hybrid local/global search strategy, and evaluate it on four long-term benchmarks and one short-term benchmark. The central theoretical result is Eq. (13), which expresses the change in the model output as a sum of a 'perfect model update' term and a 'model decay' term involving the squared norm of the gradient of the model output.

Significance. If the theoretical derivation were valid, the paper would provide a principled explanation for a well-known but poorly understood phenomenon in long-term tracking, and it would motivate concrete design choices for update policies. The empirical study of how different tracking challenges contribute to accuracy drop over extended videos is useful, and the proposed DRN with hybrid search shows promising results on several benchmarks. The paper also introduces a new long-term dataset (YouTubeLong) and a long-OTB protocol, which are potentially valuable resources. However, the central theoretical claim is not established because the derivation of Eq. (13) contains an unjustified algebraic step, as detailed below. The empirical contributions are real, but they are framed as consequences of a theory that does not hold as stated, which substantially weakens the paper's contribution.

major comments (4)
  1. [Section 2, Eq. (13)] The step from Eqs. (10) and (12) to Eq. (13) is not valid. Combining the parameter update φ_{t+1}−φ_t = −2ηE[(f_{i,t}−y*_i)∇_φ f_{i,t}] + 2ηE[δ_i∇_φ f_{i,t}] with the model change Δf_{i,t} = ∇_φ f_{i,t}·(φ_{t+1}−φ_t) yields Δf_{i,t} = −2η E[(f_{j,t}−y*_j)(∇_φ f_{i,t}·∇_φ f_{j,t})] + 2η E[δ_j(∇_φ f_{i,t}·∇_φ f_{j,t})]. The paper replaces the dot product ∇_φ f_{i,t}·∇_φ f_{j,t} with ‖∇_φ f_{i,t}‖², which would require ∇_φ f_{j,t} = ∇_φ f_{i,t} for every sample j in the expectation. This is not true for a general nonlinear tracker, nor even for a linear model f = φ^T x, where the cross terms x_i^T x_j (j≠i) do not vanish. Consequently, Eq. (13) is not a valid derivation of model decay, and the claim that the tracker model offshoots its predictions by a quantity linearly proportional to past errors is unproven. This is the central theoretical result of the paper, so the theory collapses without it.
  2. [Section 2, Eq. (10)] The decomposition into a 'perfect parameter update' and a 'parameter bias' assumes that the label noise δ_i is zero-mean and independent of the model state. In visual tracking, prediction errors are typically systematic and state-dependent: a tracker that has drifted will produce errors that are correlated with the current model parameters, and these errors are not well described by an additive zero-mean noise model. The assumption is introduced without discussion or justification, and it is essential for the clean split in Eq. (10). At minimum, the paper should state the assumptions explicitly and discuss their scope.
  3. [Section 4.1, Eq. (14)] Equation (14) is obtained by rearranging Eq. (13) and introducing the weight ω_t = 1 − δ_{i,t}/(f_{i,t}−y*_i). Since Eq. (13) is not valid, the justification for the decay recognition network as an estimator of ω_t is likewise unsupported. Even if Eq. (13) were valid, the interpretation of ω_t∈{0,1} as a binary weight that simply selects a fraction of frames for updating is a further assumption: the term (f_{i,t}−y*_i) also appears in the 'perfect update' part, and setting ω_t=0 for some frames discards the contribution of that term as well. The paper should clarify what exactly the DRN is approximating and how the update policy follows from Eq. (14).
  4. [Section 3.2, Fig. 2] The Long OTB protocol repeats the same video frames forward and backward, and the paper attributes the observed decrease in AUC over repetitions to model decay. However, with repeated frames, a tracker that updates its model can overfit to the specific appearances in the repeated sequence, causing a drop in accuracy that is not necessarily due to adverse prediction errors adding bias. The paper does not provide a control condition (e.g., a tracker that updates on the same sequence with ground-truth boxes instead of predictions) to separate overfitting from model decay. The conclusions drawn from Fig. 2 about the contribution of each tracking challenge to model decay are therefore not fully supported.
minor comments (5)
  1. [Section 2, Eqs. (5)–(6)] The notation is inconsistent: Eq. (5) uses E[(y_i−f_{i,t})²], while Eq. (6) writes E[t∇_φ f_t] and E[y_t∇_φ f_t]; the subscripts should be aligned to avoid confusion.
  2. [Section 6.1, Datasets] The text states that YouTubeLong is 'sparesely annotated every 25 frames' and then immediately says 'sparsely every 100 frames'; these numbers are contradictory.
  3. [Table 3] The caption refers to 'Tables 3' but only one table is shown; also the column header 'OxUvA' is misspelled (should be 'OxUva'), and the TPR metric mentioned in Section 6.1 is not reported anywhere.
  4. [Section 7, Conclusion] The word 'unvoidable' should be 'unavoidable'; also, the abstract claims 'superior accuracy and robustness' while the paper admits that on VOT2018 long-term LT-SINT is behind some state-of-the-art trackers; the claims should be moderated to match the reported results.
  5. [Section 4.2, DRN training] The DRN is trained on tracks produced by SINT with global search on ALOV. The paper does not discuss whether the DRN would transfer to other trackers or to substantially different appearance changes; since the paper claims the idea is general, a brief discussion of this limitation would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the model-decay decomposition is an attempted mathematical derivation, and the empirical evaluation is run on external benchmarks rather than on data used to fit the model.

full rationale

The central theoretical claim is Eq. (13), which is presented as a consequence of the parameter update in Eq. (10) via a Taylor expansion. This is not an empirical prediction fitted from test data; it is an algebraic decomposition of the update dynamics. Even though the step from Eq. (10) to Eq. (13) is mathematically questionable, circularity would require the claimed result to be equivalent to its input by construction, which is not the case here. The decay recognition network is trained offline on tracks generated by SINT on ALOV and then evaluated on UA V20L, YouTubeLong, OxUvA, and VOT2018 long-term, so no fitted parameter is renamed as a prediction. The paper cites the authors' own SINT and OxUvA work, but those citations provide the base tracker and benchmark, not the justification for the model-decay concept. The model-decay term is derived from the loss and the noise model, not assumed from a self-citation. Therefore no load-bearing circular step is identifiable, and the paper is self-contained against external benchmarks for its empirical claims.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The core 'model decay' equation (Eq. 13) depends on the zero-mean noise assumption and the incorrect gradient simplification, so the free parameters and domain assumptions listed here carry the theoretical weight. The DRN is the main invented component; it is trained on data and evaluated on external benchmarks.

free parameters (2)
  • global search interval T = 15
    Interval between global re-detections, selected in Section 6.2 by evaluating several intervals; T=15 gave real-time speed without accuracy loss.
  • DRN update threshold = unspecified
    The LSTM output threshold above which an update is performed is called 'highly conservative' but no number is given, which affects the model-update rate and the resulting tracking accuracy.
assumptions (3)
  • domain assumption Prediction errors are zero-mean Gaussian noise: y_i = y*_i + δ_i with δ_i ~ N(0, σ_i^2)
    This assumption in Eq. (8) is used to separate the 'perfect update' from the bias. In practice, lost or mistaken targets produce strongly biased errors, so the zero-mean assumption is questionable.
  • domain assumption The tracker model is updated by gradient descent on a squared-error loss over all previous frames
    The derivation from Eq. (5) to Eq. (10) assumes this update form. Many trackers (e.g., correlation filters) use closed-form or different optimization rules.
  • ad hoc to paper The change in model output after an update is given by the simplified form in Eq. (13) with ||∇_φ f||^2
    The transition from Eq. (10) to Eq. (13) requires the dot product of the gradient with an expected gradient to equal the expectation of the squared gradient; this is not generally true and is the main validity gap in the theory.
invented entities (1)
  • Decay recognition network (DRN) independent evidence
    purpose: An LSTM-based binary classifier that predicts whether a candidate model update would add bias, and gates updates to reduce model decay.
    Its performance is measured on tracking benchmarks (UA V20L, YouTubeLong, OxUvA), providing a falsifiable handle outside the derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Decay in Long-Term Tracking." pith.science (2026). https://pith.science/paper/J463SWKL

@misc{pith2026190801603,
  author       = {Pith},
  title        = {Pith review of: Model Decay in Long-Term Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J463SWKL}},
  note         = {Machine review of arXiv:1908.01603}
}
read the original abstract

Updating the tracker model with adverse bounding box predictions adds an unavoidable bias term to the learning. This bias term, which we refer to as model decay, offsets the learning and causes tracking drift. While its adverse affect might not be visible in short-term tracking, accumulation of this bias over a long-term can eventually lead to a permanent loss of the target. In this paper, we look at the problem of model bias from a mathematical perspective. Further, we briefly examine the effect of various sources of tracking error on model decay, using a correlation filter (ECO) and a Siamese (SINT) tracker. Based on observations and insights, we propose simple additions that help to reduce model decay in long-term tracking. The proposed tracker is evaluated on four long-term and one short term tracking benchmarks, demonstrating superior accuracy and robustness, even in 30 minute long videos.

Figures

Figures reproduced from arXiv: 1908.01603 by the authors.

Figure 1
Figure 1. Predictions from ECO [6] on an artificially ex￾tended video created from OTB50 data (red box: tracker prediction, yellow box: ground truth prediction). Model de￾cay is prevalent here although the appearance variation re￾mains intact. Due to heavy updating involved, model decay is noticeable from very early stages itself, even for clearly visible target objects moving slowly . half an hour. When considering the pract… view at source ↗
Figure 2
Figure 2. The effect of the 11 sources of tracking difficulty [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Running ECO, SINT, and LT-SINT on eight Long OTB videos, four videos with significant model decay ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 37 canonical work pages

  1. [1]

    Bertinetto, J

    L. Bertinetto, J. Valmadre, S. Golodetz, O. Miksik, and P. H. S. Torr. Staple: Complementary learners for real-time tracking. In CVPR, June 2016. 8

  2. [2]

    Bertinetto, J

    L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr. Fully-convolutional siamese networks for object tracking. In ECCV VOT workshop, 2016. 4, 7, 8

  3. [3]

    D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y . M. Lui. Visual object tracking using adaptive correlation filters. In CVPR, 2010. 7

  4. [4]

    Danelljan, G

    M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg. Eco: Efficient convolution operators for tracking. In CVPR, 2017. 4, 7, 8

  5. [5]

    Danelljan, G

    M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Fels- berg. Accurate scale estimation for robust visual tracking. In BMVC, 2014. 4, 7

  6. [6]

    Danelljan, G

    M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg. Adaptive decontamination of the training set: A unified for- mulation for discriminative visual tracking. In CVPR, June

  7. [7]

    Danelljan, G

    M. Danelljan, G. H ¨ager, F. Khan, and M. Felsberg. Learning spatially regularized correlation filters for visual tracking. In ICCV, 2015. 7, 8

  8. [8]

    Danelljan, A

    M. Danelljan, A. Robinson, F. S. Khan, and M. Felsberg. Beyond correlation filters: learning continuous convolution operators for visual tracking. In ECCV, 2016. 7

Show all 38 references
  1. [9]

    Danelljan, F

    M. Danelljan, F. Shahbaz Khan, M. Felsberg, and J. van de Weijer. Adaptive color attributes for real-time visual track- ing. In CVPR, 2014. 7

  2. [10]

    H. K. Galoogahi, A. Fagg, and S. Lucey. Learning background-aware correlation filters for visual tracking. In ICCV, 2017. 8

  3. [11]

    S. Hare, A. Saffari, and P. H. Torr. Struck: Structured output tracking with kernels. In ICCV, 2011. 4

  4. [12]

    J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High- speed tracking with kernelized correlation filters. TPAMI,

  5. [13]

    Z. Hong, Z. Chen, C. Wang, X. Mei, D. Prokhorov, and D. Tao. Multi-store tracker (muster): A cognitive psychol- ogy inspired approach to object tracking. In CVPR, 2015. 8

  6. [14]

    Kalal, K

    Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learning- detection. TPAMI, 34(7):1409–1422, 2010. 6, 8

  7. [15]

    Kiani Galoogahi, T

    H. Kiani Galoogahi, T. Sim, and S. Lucey. Correlation filters with limited boundaries. In CVPR, 2015. 7

  8. [16]

    Kristan, A

    M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pfugfelder, L. C. Zajc, T. V ojir, G. Bhat, A. Lukezic, A. Eldesokey, G. Fernandez, and et al. The sixth visual ob- ject tracking vot2018 challenge results, 2018. 6, 7

  9. [17]

    Kristan, J

    M. Kristan, J. Matas, A. Leonardis, M. Felsberg, L. Ce- hovin, G. Fernandez, T. V ojir, G. Hager, G. Nebehay, and R. Pflugfelder. The visual object tracking vot2015 challenge results. In ICCV VOT workshop, 2015. 1, 7

  10. [18]

    B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan. SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks. In CVPR, 2019. 6, 7

  11. [19]

    B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu. High performance visual tracking with siamese region proposal network. InThe IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2018. 7

  12. [20]

    Ma, J.-B

    C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang. Hierarchical convolutional features for visual tracking. In ICCV, 2015. 7

  13. [21]

    C. Ma, X. Yang, C. Zhang, and M.-H. Yang. Long-term correlation tracking. In CVPR, 2015. 5, 6, 8

  14. [22]

    Milan, L

    A. Milan, L. Leal-Taix ´e, I. D. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object track- ing. CoRR, 2016. 6

  15. [23]

    Moudgil and V

    A. Moudgil and V . Gandhi. Long-term visual object tracking benchmark. arXiv preprint arXiv:1712.01358, 2017. 6

  16. [24]

    Mueller, N

    M. Mueller, N. Smith, and B. Ghanem. A benchmark and simulator for uav tracking. In ECCV, 2016. 6, 7

  17. [25]

    M ¨uller, A

    M. M ¨uller, A. Bibi, S. Giancola, S. Al-Subaihi, and B. Ghanem. Trackingnet: A large-scale dataset and bench- mark for object tracking in the wild. In ECCV, 2018. 6

  18. [26]

    Nair and G

    V . Nair and G. E. Hinton. Rectified linear units improve re- stricted boltzmann machines. In ICML, 2010. 7

  19. [27]

    Nam and B

    H. Nam and B. Han. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, 2016. 4, 6, 8

  20. [28]

    Y . Qi, S. Zhang, L. Qin, H. Yao, Q. Huang, J. Lim, and M.-H. Yang. Hedged deep tracking. In CVPR, 2016. 7

  21. [29]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 7

  22. [30]

    A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah. Visual tracking: an experimental survey. TPAMI, 36(7):1442–1468, 2014. 1, 4, 6, 7

  23. [31]

    J. S. Supancic and D. Ramanan. Self-paced learning for long-term tracking. In CVPR, 2013. 6, 8

  24. [32]

    R. Tao, E. Gavves, and A. W. M. Smeulders. Siamese in- stance search for tracking. In CVPR, 2016. 3, 4, 5, 7, 8

  25. [33]

    Valmadre, L

    J. Valmadre, L. Bertinetto, J. F. Henriques, R. Tao, A. Vedaldi, A. W. M. Smeulders, P. H. S. Torr, and E. Gavves. Long-term tracking in the wild: A benchmark. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I...

  26. [34]

    Valmadre, L

    J. Valmadre, L. Bertinetto, J. F. Henriques, A. Vedaldi, and P. H. Torr. End-to-end representation learning for correlation filter based tracking. 2017. 8

  27. [35]

    Y . Wu, J. Lim, and M.-H. Yang. Online object tracking: A benchmark. In CVPR, 2013. 4

  28. [36]

    Y . Wu, J. Lim, and M.-H. Yang. Object tracking benchmark. TPAMI, 37(9):1834–1848, 2015. 1, 7, 8

  29. [37]

    G. Zhu, F. Porikli, and H. Li. Beyond local search: Track- ing objects everywhere with instance-specific proposals. In CVPR, 2016. 6, 8

  30. [38]

    Z. Zhu, Q. Wang, B. Li, W. Wu, and W. Hu. Distractor-aware Siamese Networks for Visual Object Tracking. In CVPR,

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.