{"id":"9711f8a9-0a27-4386-a084-30d20a23fe4d","arxiv_id":"1908.01603","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Model decay, an accumulated bias from updating on past prediction errors, is formally proposed as a cause of long-term tracking drift, and the LT-SINT tracker with a decay recognition network and hybrid search is presented to reduce it.","lead":"This paper proposes a mathematical explanation for why video trackers gradually fail on long videos: each update of the tracker's model carries a bias from past mistakes, and these biases add up. The authors then build a Siamese tracker with a learned 'decay recognition' gate and periodic global re-search, and test it on long-term tracking benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (13) does not follow from Eq. (10): the dot product of gradients is replaced by a squared norm, ignoring cross-sample terms; the central derivation of model decay is invalid.","rationale":"The paper aims to give a mathematical foundation for the intuitive notion that self-updating trackers accumulate bias. The central claim is that the tracker model output evolves according to Eq. (13), with a term proportional to past errors. The reader's critique targets the step from Eq. (10) to Eq. (13), and my analysis confirms that this step is mathematically unjustified. When the parameter update is an expectation over all past samples, the change in the output for a specific frame involves a dot product between the gradient at that frame and the gradients of all frames in the expectation. The paper simply writes the squared norm of the gradient, which would require the gradients to be parallel or the expectation to collapse to a single sample. Neither condition holds in general, as demonstrated by a two-sample linear model. This is not a minor technical slip: Eq. (13) is the formal statement of 'model decay', and without it the paper's theoretical explanation of long-term tracking drift is unsupported. The empirical tracker LT-SINT may still have merit, but the paper's primary novelty—the mathematical analysis—fails. I therefore see no reason to alter the reader's rejection. The proposed concrete test would settle the matter definitively by showing the exact left-hand and right-hand sides of Eq. (13) differ in a simple, reproducible setting.","tokens_in":12655,"tokens_out":5297,"duration_ms":43822,"concrete_test":"Implement a minimal two-sample linear model: f_i = φ^T x_i, with fixed x_1, x_2, y_1 = y*_1, y_2 = y*_2 + δ, and compute φ_{t+1} = φ_t - 2η[(f_1 - y_1)x_1 + (f_2 - y_2)x_2]. Then compute Δf_1 = x_1^T(φ_{t+1} - φ_t) and compare with the right-hand side of Eq. (13) for i=1. For generic x_1, x_2 with x_1^T x_2 ≠ ||x_1||^2 and δ ≠ 0, the two expressions differ (the exact Δf_1 contains the cross term -2η (f_2 - y_2) x_1^T x_2). If they differ, the derivation of Eq. (13) from Eq. (10) is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core derivation of model decay is Eq. (13), which states f_{i,t+1} - f_{i,t} = -2η E[(f_i - y*_i)||∇_φ f_i||^2] + 2η E[δ_i||∇_φ f_i||^2]. To obtain this from the parameter update Eq. (10), one must compute Δf_i = ∇_φ f_i · Δφ = -2η ∇_φ f_i · E[(f - y)∇_φ f] = -2η E[(f - y)(∇_φ f_i · ∇_φ f)]. The paper replaces ∇_φ f_i · ∇_φ f with ||∇_φ f_i||^2. This replacement is valid only if the gradient of the model output at frame i is identical to the gradient appearing in the expectation for every sample, i.e., ∇_φ f_j = ∇_φ f_i for all j, or if the update uses only sample i. Neither is the case for the stochastic gradient descent over the history specified by Eq. (7). For a generic nonlinear tracker, and even for a linear model f = φ^T x, the cross terms Σ_{j≠i} (f_j - y_j) x_i^T x_j do not vanish and are not accounted for. Therefore Eq. (13) is not a valid derivation; the claimed property that model decay is linearly proportional to past errors in the model output is unproven. The empirical LT-SINT results may be useful, but the theoretical contribution collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the phenomenon of model decay in long-term visual object tracking. It presents a mathematical analysis of the learning dynamics of tracker models, claiming that any tracker updating on its own predictions accumulates a bias term—model decay—that is linearly proportional to past prediction errors. Based on this analysis, the authors propose LT-SINT, a Siamese tracker augmented with a decay recognition network (DRN) and a hybrid local/global search strategy, and evaluate it on four long-term benchmarks and one short-term benchmark. The central theoretical result is Eq. (13), which expresses the change in the model output as a sum of a 'perfect model update' term and a 'model decay' term involving the squared norm of the gradient of the model output.","tokens_in":12989,"tokens_out":5269,"duration_ms":50162,"significance":"If the theoretical derivation were valid, the paper would provide a principled explanation for a well-known but poorly understood phenomenon in long-term tracking, and it would motivate concrete design choices for update policies. The empirical study of how different tracking challenges contribute to accuracy drop over extended videos is useful, and the proposed DRN with hybrid search shows promising results on several benchmarks. The paper also introduces a new long-term dataset (YouTubeLong) and a long-OTB protocol, which are potentially valuable resources. However, the central theoretical claim is not established because the derivation of Eq. (13) contains an unjustified algebraic step, as detailed below. The empirical contributions are real, but they are framed as consequences of a theory that does not hold as stated, which substantially weakens the paper's contribution.","major_comments":[{"comment":"The step from Eqs. (10) and (12) to Eq. (13) is not valid. Combining the parameter update φ_{t+1}−φ_t = −2ηE[(f_{i,t}−y*_i)∇_φ f_{i,t}] + 2ηE[δ_i∇_φ f_{i,t}] with the model change Δf_{i,t} = ∇_φ f_{i,t}·(φ_{t+1}−φ_t) yields Δf_{i,t} = −2η E[(f_{j,t}−y*_j)(∇_φ f_{i,t}·∇_φ f_{j,t})] + 2η E[δ_j(∇_φ f_{i,t}·∇_φ f_{j,t})]. The paper replaces the dot product ∇_φ f_{i,t}·∇_φ f_{j,t} with ‖∇_φ f_{i,t}‖², which would require ∇_φ f_{j,t} = ∇_φ f_{i,t} for every sample j in the expectation. This is not true for a general nonlinear tracker, nor even for a linear model f = φ^T x, where the cross terms x_i^T x_j (j≠i) do not vanish. Consequently, Eq. (13) is not a valid derivation of model decay, and the claim that the tracker model offshoots its predictions by a quantity linearly proportional to past errors is unproven. This is the central theoretical result of the paper, so the theory collapses without it.","section":"Section 2, Eq. (13)"},{"comment":"The decomposition into a 'perfect parameter update' and a 'parameter bias' assumes that the label noise δ_i is zero-mean and independent of the model state. In visual tracking, prediction errors are typically systematic and state-dependent: a tracker that has drifted will produce errors that are correlated with the current model parameters, and these errors are not well described by an additive zero-mean noise model. The assumption is introduced without discussion or justification, and it is essential for the clean split in Eq. (10). At minimum, the paper should state the assumptions explicitly and discuss their scope.","section":"Section 2, Eq. (10)"},{"comment":"Equation (14) is obtained by rearranging Eq. (13) and introducing the weight ω_t = 1 − δ_{i,t}/(f_{i,t}−y*_i). Since Eq. (13) is not valid, the justification for the decay recognition network as an estimator of ω_t is likewise unsupported. Even if Eq. (13) were valid, the interpretation of ω_t∈{0,1} as a binary weight that simply selects a fraction of frames for updating is a further assumption: the term (f_{i,t}−y*_i) also appears in the 'perfect update' part, and setting ω_t=0 for some frames discards the contribution of that term as well. The paper should clarify what exactly the DRN is approximating and how the update policy follows from Eq. (14).","section":"Section 4.1, Eq. (14)"},{"comment":"The Long OTB protocol repeats the same video frames forward and backward, and the paper attributes the observed decrease in AUC over repetitions to model decay. However, with repeated frames, a tracker that updates its model can overfit to the specific appearances in the repeated sequence, causing a drop in accuracy that is not necessarily due to adverse prediction errors adding bias. The paper does not provide a control condition (e.g., a tracker that updates on the same sequence with ground-truth boxes instead of predictions) to separate overfitting from model decay. The conclusions drawn from Fig. 2 about the contribution of each tracking challenge to model decay are therefore not fully supported.","section":"Section 3.2, Fig. 2"}],"minor_comments":[{"comment":"The notation is inconsistent: Eq. (5) uses E[(y_i−f_{i,t})²], while Eq. (6) writes E[t∇_φ f_t] and E[y_t∇_φ f_t]; the subscripts should be aligned to avoid confusion.","section":"Section 2, Eqs. (5)–(6)"},{"comment":"The text states that YouTubeLong is 'sparesely annotated every 25 frames' and then immediately says 'sparsely every 100 frames'; these numbers are contradictory.","section":"Section 6.1, Datasets"},{"comment":"The caption refers to 'Tables 3' but only one table is shown; also the column header 'OxUvA' is misspelled (should be 'OxUva'), and the TPR metric mentioned in Section 6.1 is not reported anywhere.","section":"Table 3"},{"comment":"The word 'unvoidable' should be 'unavoidable'; also, the abstract claims 'superior accuracy and robustness' while the paper admits that on VOT2018 long-term LT-SINT is behind some state-of-the-art trackers; the claims should be moderated to match the reported results.","section":"Section 7, Conclusion"},{"comment":"The DRN is trained on tracks produced by SINT with global search on ALOV. The paper does not discuss whether the DRN would transfer to other trackers or to substantially different appearance changes; since the paper claims the idea is general, a brief discussion of this limitation would be helpful.","section":"Section 4.2, DRN training"}],"recommendation":"reject","confidential_remarks":"The core theoretical derivation (Eq. 13) is invalid, and the paper's central claim about model decay being linearly proportional to past errors is not supported. While the empirical contributions—the Long OTB protocol, the YouTubeLong dataset, and the DRN-based tracker—could form the basis of a useful empirical paper, the current manuscript's identity as a theoretical analysis is undermined. A rejection is appropriate unless the authors can provide a corrected theoretical framework that either fixes the derivation or substantially revises the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know about this paper is that the empirical package is more interesting than the theory it is wrapped in. The \"model decay\" derivation in Section 2 has a load-bearing step that does not hold. Going from Eq. (10) to Eq. (13), the paper computes the change in the model output at frame i as if the parameter gradient in the update were the gradient at frame i alone. In reality, Δf_i = ∇_φ f_i · Δφ, so you get dot products ∇_φ f_i · ∇_φ f_j under the expectation. The cross terms j ≠ i do not vanish for a generic nonlinear tracker, or even for a linear model. Eq. (13) is therefore not a valid consequence of Eq. (10). This matters because the central claim that model decay is \"linearly proportional to past errors\" is exactly what Eq. (13) is supposed to establish. As written, the theory collapses.\n\nThat said, there is real value in the rest of the paper. The Long OTB replay protocol is a simple, cheap way to expose drift in a controlled setting, and the experiments on it are informative. The LT-SINT tracker combines global search with an LSTM \"decay recognition network\" in a sensible way, and the ablations in Table 2 (no updates, every frame, similarity threshold, LSTM gate) actually isolate where the gain comes from. The four-dataset evaluation, including the very long YouTubeLong sequences, is more than most tracking papers ship. The authors are also honest about the VOT-LT result being behind heavier architectures, which I respect.\n\nOther soft spots: the Gaussian-noise assumption on δ_i is hard to swallow, because δ_i is the tracker's own prediction error, not exogenous noise; and the \"bias\" term in Eq. (10) is just the standard gradient of the self-training loss, so the novelty is mostly in the interpretation. The DRN is trained on tracks from SINT and tested on other benchmarks, so there is a mild domain gap, but the held-out setting is reasonable. No code or data is released, which is a pity for a paper whose main diagnostic is a new protocol.\n\nWho should read this: anyone building long-term trackers will get a useful baseline and a practical warning about self-updates. The theoretical section should be skipped or rewritten. I would send it to peer review, because the empirical contribution is solid enough to withstand a required rewrite of the theory.","headline":"The empirical long-term tracker is worth a look, but the central derivation of 'model decay' rests on a wrong linearization, so the theory should not be taken as proven.","tokens_in":13507,"tokens_out":3398,"would_cite":false,"duration_ms":34905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Updating a tracker on its own predictions adds an unavoidable bias—model decay—that grows linearly with past errors, and this accumulated bias is what makes long-term trackers drift and lose the target.","keywords":["model decay","long-term tracking","visual object tracking","online learning bias","Siamese trackers","correlation filters","self-training drift","tracking benchmarks"],"falsifier":"Run an online-updating tracker on a long video with known ground truth, and at every update record the prediction error $\\delta_i$, the model-output gradient $\\nabla_\\varphi f_{i,t}$, and the actual change in the model's predicted box $f_{i,t+1}-f_{i,t}$. The theory predicts that the component of this change beyond the 'perfect update' equals $2\\eta\\,\\mathbb{E}[\\delta_{i,t}\\|\\nabla_\\varphi f_{i,t}\\|^2]$. If the measured shift does not match this linear, gradient-squared-weighted form—for instance, if it depends on other quantities or does not scale with $2\\eta$—then the model-decay term as derived is not the mechanism driving drift.","tokens_in":12452,"feed_emoji":"🎯","tokens_out":15770,"duration_ms":133495,"temperature":0.7,"pith_summary":"Visual object trackers that keep updating their model using their own bounding-box predictions as training labels are adding an unavoidable bias to the learning, a bias the authors call model decay. The paper derives this bias from the standard gradient-update equations and shows that the model output shifts by a quantity linearly proportional to past prediction errors. Because the bias accumulates recursively over time, short tracking videos hide it while long videos expose it as drift and eventual target loss. The authors show empirically that a Siamese tracker whose updates are gated by a separately trained 'decay recognition network' reduces this decay and stays accurate on videos up to 30 minutes long. If the derivation is right, long-term tracking failures are not a bug in any single tracker but a mathematical property of self-training.","feed_headline":"Trackers that learn from their own guesses are built to drift","feed_subtitle":"The bias grows linearly with past prediction errors, explaining why 30-minute videos break trackers.","key_machinery":"The key machinery is the linearized model-dynamics identity that decomposes the change in a tracker's model output after an online gradient step into a perfect-update term and a model-decay term $2\\eta\\mathbb{E}[\\delta_{i,t}\\|\\nabla_\\varphi f_{i,t}\\|^2]$, which is linear in the past prediction errors $\\delta_{i,t}$ and weighted by the squared sensitivity of the model output to its parameters. This identity turns the vague intuition that 'bad updates cause drift' into a concrete, recursive quantity that accumulates over time and can be measured. The paper's practical counterpart is the 'decay recognition network', an LSTM trained on Siamese similarity maps that decides whether a candidate update is likely to add bias, thereby approximating a gate on whether the model should be updated at all.","core_discovery":"The central discovery is that the online update of any tracker model trained on its own outputs splits into two components: a perfect update that corrects the model toward the true target box, and a bias term proportional to the noise in past predictions. Concretely, after one gradient step the model output at a frame $i$ changes as $f_{i,t+1}=f_{i,t}-2\\eta\\mathbb{E}[(f_{i,t}-y^*_i)\\|\\nabla_\\varphi f_{i,t}\\|^2]+2\\eta\\mathbb{E}[\\delta_{i,t}\\|\\nabla_\\varphi f_{i,t}\\|^2]$, where the first term is the 'perfect model update' and the second term, the 'model decay', offshoots the model by a quantity linearly proportional to past errors $\\delta_i$. Because this decay term accumulates recursively, any tracker that keeps updating on imperfect self-predictions drifts, and the drift is a mathematical consequence of self-training rather than a failure of a particular algorithm. The authors demonstrate the effect on correlation-filter and Siamese trackers, and use the insight to build LT-SINT, which gates updates with a learned decay-recognition network and global search, reporting that it avoids significant decay even in 30-minute videos.","pith_inferences":["Beyond the paper: the same bias term should appear in any online-learning system that trains on its own outputs, such as self-training language models or bootstrapped reinforcement learning, so the linear-growth formula gives a concrete way to test for 'model collapse' or policy drift in those settings.","Beyond the paper: because the decay term is weighted by $\\|\\nabla_\\varphi f_{i,t}\\|^2$, reducing a model's output sensitivity—via regularization, smaller learning rates, or flatter minima—should slow decay even without a gating network; the paper does not test this prediction.","Beyond the paper: the decay-recognition network is a general pattern—an external meta-controller that decides when a learner should update, trained on the learner's confidence signals rather than its parameters—that could transfer to continual learning."],"forward_implications":["Any tracker that re-trains on its own predictions carries a built-in drift: the model-decay term is unavoidable under gradient updates, regardless of architecture.","The decay grows with the number of updates and the magnitude of past errors, which is why 10-to-20-second benchmarks hide the problem while 30-minute videos expose it as target loss.","A tracker that never updates, such as a pure Siamese matcher, has zero model decay but cannot adapt to appearance change; LT-SINT shows sparse, gated updates as a working middle ground.","Gating updates with a separately trained decay-recognition network improves AUC on UAV20L, YouTubeLong, and OxUvA compared with both updating every frame and never updating.","The approach also scores 59.8 AUC on the short-term OTB benchmark, close to popular short-term trackers, indicating long-term robustness does not sacrifice short-term utility."],"supporting_citations":[{"why":"Provides the ECO correlation-filter tracker used in the model-decay analysis and as a baseline.","marker":"[6]"},{"why":"Provides the Siamese instance search (SINT) tracker that LT-SINT extends and the no-update baseline.","marker":"[32]"},{"why":"Supplies the OTB-50 dataset and short-term evaluation protocol used to build Long OTB and for the short-video experiment.","marker":"[36]"},{"why":"Supplies the eleven tracking-challenge annotations used to analyze which challenges drive model decay.","marker":"[35]"},{"why":"Provides the OxUvA long-term benchmark and the true-positive-ratio evaluation metric used in experiments.","marker":"[33]"},{"why":"Provides the UAV20L long-term benchmark used in the evaluation.","marker":"[24]"},{"why":"Provides the VOT2018 long-term dataset and F-score metric used for comparison.","marker":"[16]"}],"fun_headline_variants":["Self-updates mathematically guarantee tracker drift","Bias from past errors explains long-term tracking loss","Tracker decay: the math behind losing targets in long videos","Long-term trackers fail because they learn from their own mistakes","Model decay: why trackers drift from noisy self-learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central derivation assumes that the way a tracker's output changes when its parameters change can be summarized by the squared size of that change; if that algebraic step is not valid for a given model, the claimed linear growth of model decay with past errors does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Self-updates mathematically guarantee tracker drift","Bias from past errors explains long-term tracking loss","Tracker decay: the math behind losing targets in long videos","Long-term trackers fail because they learn from their own mistakes","Model decay: why trackers drift from noisy self-learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1668,"prompt_tokens":938,"completion_tokens":730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":653}},"tokens_in":554,"tokens_out":730,"duration_ms":7857,"temperature":1.0,"reasoning_tokens":653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:07:44.516917+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an online-updating tracker on a long video with known ground truth, and at every update record the prediction error $\\delta_i$, the model-output gradient $\\nabla_\\varphi f_{i,t}$, and the actual change in the model's predicted box $f_{i,t+1}-f_{i,t}$. The theory predicts that the component of this change beyond the 'perfect update' equals $2\\eta\\,\\mathbb{E}[\\delta_{i,t}\\|\\nabla_\\varphi f_{i,t}\\|^2]$. If the measured shift does not match this linear, gradient-squared-weighted form—for instance, if it depends on other quantities or does not scale with $2\\eta$—then the model-decay term as derived is not the mechanism driving drift.","supporting_citations":[{"cited_title":"Danelljan, G","cited_arxiv_id":null,"evidence_quote":"Provides the ECO correlation-filter tracker used in the model-decay analysis and as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Siamese instance search (SINT) tracker that LT-SINT extends and the no-update baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OTB-50 dataset and short-term evaluation protocol used to build Long OTB and for the short-video experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the eleven tracking-challenge annotations used to analyze which challenges drive model decay."},{"cited_title":"Valmadre, L","cited_arxiv_id":null,"evidence_quote":"Provides the OxUvA long-term benchmark and the true-positive-ratio evaluation metric used in experiments."},{"cited_title":"Mueller, N","cited_arxiv_id":null,"evidence_quote":"Provides the UAV20L long-term benchmark used in the evaluation."},{"cited_title":"Kristan, A","cited_arxiv_id":null,"evidence_quote":"Provides the VOT2018 long-term dataset and F-score metric used for comparison."}],"review_version":1}