{"id":"39bafe86-b0ed-4e2b-a4ff-7c333e79b256","arxiv_id":"2501.03220","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ProTracker merges optical flow predictions with long-term keypoint matches through probabilistic integration, achieving the highest point-position accuracy on TAP-Vid and BADJA.","lead":"ProTracker is a new algorithm for following any point in a video across many frames, combining short-term optical flow with long-term semantic matches. It reports the most accurate point positions on several standard tracking benchmarks, especially for videos with occlusions and similar-looking regions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probabilistic fusion weights depend on unreported correlation parameter p and arbitrary keypoint variance; sensitivity to these could overturn claimed SOTA margins.","rationale":"The paper's strongest claim is empirical: it achieves the highest delta_x_avg on four benchmark configurations, and attributes this to the probabilistic integration of flow and long-term keypoints. For that claim to hold, the fusion rule must be well-specified and its parameters must not be silently chosen to favor the reported numbers. The load-bearing assumption is that Eq. (6) and Eq. (11) describe a principled or at least fixed combination rule. The paper violates this by omitting p, a parameter that directly scales the flow variance and thereby changes the weight given to the keypoint observation. To see how sensitive the result could be, note the ablation where probabilistic integration is replaced by lowest-sigma selection changes delta_x_avg by 0.7 (First) and 1.8 (Strided) relative to the full model. If different p values induce shifts of that size, the 0.2-0.7 point margins over CoTracker3 and BootsTAPIR could change direction. The keypoint variance sigma_key = 1 is equally arbitrary; only the ratio sigma_flow^2/sigma_key^2 matters, so the pair (p, sigma_key) forms an under-specified two-parameter family that governs the final prediction. The orthogonal-Jacobian assumption in Eq. (2)-(3) is a stated simplification, and although it is technically questionable for non-rigid motion, it is transparent and less concerning than a completely hidden parameter. The paper otherwise provides valuable implementation details (thresholds, time intervals, outlier distance), and the ablations show all components contribute positively, which is independent support that the broad design is effective. But the exact fusion mechanism is central to the claimed novelty, and without p the method cannot be reproduced. The proposed sweep over p and sigma_key would settle whether the reported SOTA is robust or an artifact of unreported tuning. This is consistent with the reader's conditional verdict; I am not moving the verdict, but I am sharpening the reader's weakest assumption into a concrete falsifiable check.","tokens_in":14150,"tokens_out":5729,"duration_ms":54080,"concrete_test":"Re-run ProTracker on TAP-Vid DA VIS First and Strided (and BADJA if feasible) with p in {0, 0.25, 0.5, 0.75, 0.9} and sigma_key in {0.5, 1, 2}, keeping all other settings identical, and report delta_x_avg and AJ. If the ranking versus CoTracker3 and BootsTAPIR changes, or the spread across p exceeds roughly 1 delta_x_avg point, the headline SOTA result is tied to an unreported knob; if performance is flat, publish the p value and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim rests on the probabilistic fusion in Eq. (6) and Eq. (11). Eq. (6) inflates the integrated flow variance by a factor sqrt((N-1)p+1), where p is a constant correlation coefficient never assigned a value anywhere in the paper or supplement. Eq. (10) fixes the keypoint variance sigma_key = 1 without justification. These two quantities set the relative weight between flow and keypoint in Eq. (11) and together determine the final position estimate. Because p is unreported, the reported delta_x_avg numbers are not reproducible. Moreover, the mean weights in Eq. (6) are inverse-variance weights valid only for independent estimates, while the variance inflation assumes constant correlation; this is internally inconsistent except in the equal-variance special case. The margins over the closest baselines are small (e.g., 0.7 on DA-VIS First, 0.2 on Kinetics First), so if the optimal p is benchmark-specific or even mildly variable, the claim that the fusion framework is the source of SOTA precision is not supported. A sensitivity check over p is required.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ProTracker is a test-time optimization point tracker that combines chained RAFT optical flow predictions with long-term keypoint correspondences from a DINO-based feature model. The pipeline first filters rough predictions with SAM2-style object masks and geometry-aware semantic features, then integrates filtered flow predictions from multiple temporal intervals in forward and backward directions using Gaussian inverse-variance weighting, and finally fuses the integrated flow estimate with high-confidence long-term keypoints through a second inverse-variance fusion. The authors evaluate on TAP-Vid DA VIS, TAP-Vid Kinetics, and BADJA, reporting state-of-the-art results among optimization-based trackers and the highest delta_x_avg among all compared methods, with component ablations and an efficiency comparison against DINO-Tracker.","tokens_in":14413,"tokens_out":4874,"duration_ms":44461,"significance":"If the reported results hold, ProTracker is a practically valuable contribution: it demonstrates that a probabilistic coupling of local optical flow and global semantic keypoints can achieve competitive or superior precision relative to supervised feed-forward trackers, without end-to-end supervised training of the whole pipeline. The paper's strengths include extensive comparisons against many recent baselines, component ablations that isolate the main algorithmic choices, and a clear efficiency advantage over DINO-Tracker. The probabilistic formulation is simple and appealing. However, the central claim that the fusion framework is the source of the SOTA precision is currently under-specified: the fusion depends on an unreported correlation coefficient and an arbitrary keypoint variance, and the independence assumption between the two fused signals is questionable given the self-supervised training signal.","major_comments":[{"comment":"The correlation coefficient p introduced in Eq. (6) is never assigned a value in the main text or the supplement, yet the variance inflation term (N-1)p+1 directly controls the relative weight of the integrated flow estimate in the final fusion Eq. (11). With the reported margins being small (e.g., 0.7 delta_x_avg on DA-VIS First and 0.2 on Kinetics First in Table 2), the conclusion that the fusion framework explains the SOTA precision requires either a specified p or a sensitivity analysis over a plausible range of p. As written, the experiments are not reproducible from the paper alone.","section":"Sec. 3.2, Eq. (6)"},{"comment":"The derivation replaces the covariance propagation J sigma^2 J^T by the scalar sum sigma^2_j + sigma^2_fji via the assumption that J is orthogonal. This is only exact for translations and small rotations; under large rotations or non-rigid motion the Jacobian is not orthogonal, so the propagated variance is incorrect. Since sigma_i enters the inverse-variance weights in Eq. (4) and Eq. (11), the approximation is not merely cosmetic. Please justify the assumption or propagate the full covariance.","section":"Sec. 3.2, Eqs. (2)-(3)"},{"comment":"The long-term keypoint model is trained with optical flow as a self-supervised signal ('we train a feature extractor ... with the optical flow as a self-supervised signal'). The flow-integrated prediction and the keypoint observation are therefore not independent, because the keypoint model has been fit to the same RAFT flow that is being integrated. Treating the two as independent Gaussian likelihoods in Eq. (11) violates the independence assumption underlying the product-of-Gaussians fusion. Please quantify the correlation or discuss how the training signal is made independent, for example by using different flow frames or withholding the target flow.","section":"Sec. 3.3, Eqs. (8)-(12)"},{"comment":"The keypoint variance is set to sigma_key = 1 without any calibration or justification. The ratio sigma_i / sigma_key determines how strongly the long-term keypoint pulls the final estimate in Eq. (11). A unit variance may be reasonable for a heatmap SoftArgMax error, but the paper should report how this value was chosen and how sensitive the final delta_x_avg is to it. Without this, the balance between flow and keypoint is under-specified.","section":"Sec. 3.3, Eq. (10)"},{"comment":"All reported numbers are point estimates without error bars or repeated-run statistics. Some headline margins are very small (e.g., Kinetics-First delta_x_avg 71.1 vs. 70.9 for CoTracker3 in Table 2), and the ablation gains in Table 3 are also single runs. To support the load-bearing claim that the full system 'achieves the highest delta_x_avg across all datasets,' please provide standard deviations over at least three runs or bootstrap confidence intervals per video.","section":"Sec. 4.2, Tables 1-3"}],"minor_comments":[{"comment":"The symbol N is not defined in the main text; it appears to be the number of integrated predictions. Please define it.","section":"Sec. 3.2, Eq. (6)"},{"comment":"The sentence 'where mu_i represents the final predicted position' should refer to mu^f_i, to avoid confusion with the preliminary refined mean in Eq. (4).","section":"Sec. 3.2, after Eq. (6)"},{"comment":"The main text states that a prediction is invalid if feature similarity to the query falls below 0.5, while the supplement says flow predictions use a threshold of 0.3 and long-term keypoints use 0.7; please make the main-text description consistent with the supplement.","section":"Sec. 3.1 vs. Supp. Sec. 2.1"},{"comment":"The notation (Oji, p) > rho should be Oji(p) > rho, and the symbol rho is reused later for the keypoint similarity threshold; please use distinct symbols to avoid ambiguity.","section":"Supp. Sec. 2.2, Eq. (1)"},{"comment":"The sentence 'our method don't require any hyperparameter tuning' should read 'does not require'.","section":"Supp. Sec. 2.1"}],"recommendation":"major_revision","confidential_remarks":"The two most serious reproducibility blockers are the missing value of p in Eq. (6) and the unspecified calibration of sigma_key in Eq. (10). Even if the authors have fixed values in their code, the manuscript should state them and provide a sensitivity analysis. The independence/circularity concern about the self-supervised keypoint model is real and should be addressed with either a formal argument or a control experiment. The paper is otherwise a solid empirical study, but the current presentation overstates the precision of the SOTA claim given the small margins and absent error bars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid engineering contribution to point tracking. The novel bit is the correlation-adjusted Gaussian fusion of chained optical flow and long-term semantic keypoints, plus the hybrid filter combining SAM2 masks and geometry-aware features. The ablations show each component earns its keep, and the reported numbers on TAP-Vid and BADJA are competitive, hitting the best delta_x_avg among optimization-based trackers.\n\nThe empirical work is extensive and the ablations isolate each piece cleanly. I believe the central claim—that the fusion framework helps—is supported by the tables, even if the margins over baselines are a few points at best.\n\nThe soft spots are in the derivation and reproducibility. Equation (6) introduces a correlation coefficient p that is never given a value anywhere in the paper or supplement. That p sets the variance inflation and therefore the weight of flow versus keypoints in Eq. (11). The keypoint variance sigma_key=1 is also fixed without justification. Without those two quantities, the reported numbers are not reproducible. A sensitivity analysis over p would address this; I'd ask for it in revision.\n\nSecond, the long-term keypoint model is trained with RAFT flow as its self-supervised signal, so the 'independent' observation in Eq. (11) is partly derived from the same flow being integrated. The paper doesn't discuss this coupling. It doesn't necessarily overturn the results, but it weakens the claim that the two sources are independent.\n\nThird, the mean weights in Eq. (4) are inverse-variance weights valid only for independent estimates, yet Eq. (6) inflates the variance with a correlation term while keeping the same weights. That's a heuristic, not a derivation. The paper should label it as such.\n\nNone of these are fatal. The pipeline is well-built and the empirical case is made. The paper deserves a serious referee, with requests for the p sensitivity check, a justification for sigma_key, and a discussion of the training coupling. I'd accept for peer review.\n\n— [Your name]","headline":"Solid engineering contribution to point tracking, but the unreported correlation coefficient p in the fusion step makes the exact numbers non-reproducible; still worth refereeing.","tokens_in":14898,"tokens_out":3763,"would_cite":true,"duration_ms":31143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProTracker claims that probabilistic integration of optical flow and long-term keypoint matches delivers state-of-the-art point-tracking accuracy, beating all compared methods on position precision across TAP-Vid and BADJA.","keywords":["point tracking","optical flow","long-term correspondence","probabilistic integration","Kalman filter","occlusion handling","TAP-Vid benchmark","test-time optimization"],"falsifier":"Run ProTracker on TAP-Vid DA VIS with $p$ fixed to 0, 0.5, and 1 and with the full flow Jacobian kept in Eq. (2); if $\\delta x_{\\text{avg}}$ does not peak near the paper's implicit $p$, or if the orthogonal-Jacobian version does not beat the full-Jacobian version, then the fusion rule as specified is not the mechanism behind the reported accuracy.","tokens_in":13983,"feed_emoji":"🎯","tokens_out":7914,"duration_ms":62058,"temperature":0.7,"pith_summary":"ProTracker makes the case that long-term dense point tracking is best framed as a probabilistic fusion problem rather than a choice between local and global trackers. The paper's claim is that chaining optical-flow predictions from multiple past frames, treating each as a Gaussian, and merging them with high-confidence long-term keypoint matches yields the precision of iterative local trackers and the occlusion robustness of global heatmap methods simultaneously. On TAP-Vid DA VIS (first and strided queries), Kinetics-first, and BADJA, the reported position accuracy, measured by $\\delta x_{\\text{avg}}$, is the highest among all methods compared, and it is state-of-the-art among optimization-based trackers. The practical payoff is a test-time tracker that recovers points after occlusion and reappearance without drifting.","feed_headline":"Probabilistic fusion of flow and keypoint matches tops TAP-Vid","feed_subtitle":"New tracker merges short-term optical flow with long-term matches, winning position accuracy on all four benchmarks.","key_machinery":"The load-bearing mechanism is the bidirectional probabilistic integration rule, which treats every prediction as an isotropic Gaussian. For a chain from frame $j$ to $i$, the mean is $\\mu_{ji} = \\mu_j + f_{ji}$ and the variance is $\\sigma_{ji}^2 = \\sigma_j^2 + \\sigma_{fji}^2$ (assuming an orthogonal flow Jacobian), and merging $N$ such predictions gives a combined variance of $\\sqrt{(N-1)p + 1 / \\sum_j 1/\\sigma_{ji}^2}$, where $p$ is a constant correlation coefficient. The joint integration then treats a long-term keypoint as an independent Gaussian with $\\sigma = 1$ and combines it with the flow-integrated estimate by inverse-variance weighting (Eq. 11). A hybrid filter removes predictions before integration, using object-level segmentation masks and geometry-aware features with thresholds 0.7 for keypoints and 0.3 for flow.","core_discovery":"The central claim is that the fusion framework itself, not any single component, produces the accuracy gain. Rough optical-flow predictions chained from frames at intervals $\\{\\infty, 1, 2, 4, 8, 16, 32\\}$ are first filtered by an object-level mask and a geometry-aware feature similarity check, then merged into one Gaussian estimate by inverse-variance weighting with a constant correlation coefficient $p$, and the same update combines this flow estimate with long-term keypoint observations derived from a refined DINO heatmap. The author states that 'our approach achieves the highest $\\delta x_{\\text{avg}}$ across all datasets, demonstrating superior precision in tracking visible points.' The mechanism also runs backward from the last frame to recover points the forward pass marked occluded, and it re-localizes points that disappear and reappear.","pith_inferences":["The constant correlation coefficient $p$ is never given a value in the paper, so measuring it empirically from chained-flow error correlations would make the method fully reproducible and likely change the optimal balance between flow and keypoints.","The orthogonality assumption on the flow Jacobian is a known weak point for large rotations and non-rigid deformation; testing the full-Jacobian update would reveal how much of the gain depends on that simplification.","A natural extension is to apply the same probabilistic integration to other backbone pairs, for instance replacing RAFT with a learned long-range flow model, to test whether the fusion rule itself generalizes or whether the gain is tied to the specific flow and keypoint models used here."],"forward_implications":["Points that disappear and later reappear can be re-localized by the long-term keypoint branch, after which optical flow resumes tracking, recovering trajectory segments that pure flow chaining would lose.","Because the fused variance shrinks as more valid predictions accumulate, accuracy should improve with longer integration spans without retraining, up to the limit where stale flow predictions violate the Gaussian error model.","The reported results imply that a test-time optimization tracker can beat supervised feed-forward trackers on position precision by combining cheap short-term flow with sparse long-term matches, shifting the design trade-off away from larger networks.","The same fusion update applies to any noisy source of point predictions, so swapping in a different optical-flow or correspondence model should preserve the robustness gains as long as the filter and the variance estimates are retained."],"supporting_citations":[{"why":"Supplies the RAFT optical flow and its uncertainty map, the primary short-term motion signal.","marker":"[47]"},{"why":"MFT provides the chaining schedule {∞,1,2,4,8,16,32} and the direct flow-chaining baseline that ProTracker extends.","marker":"[33]"},{"why":"DINO-Tracker contributes the test-time-trained delta-DINO feature extractor and heatmap refiner used for long-term keypoints.","marker":"[49]"},{"why":"TAP-Vid defines the benchmark, the δx_avg/OA/AJ metrics, and the DA VIS and Kinetics evaluation protocols.","marker":"[12]"},{"why":"Omnimotion supplies the Kinetics-100 subset and the optimization-based canonical-volume baseline for comparison.","marker":"[50]"},{"why":"Provides the object-level masks used in the hybrid filter to discard out-of-object flow predictions.","marker":"[37]"},{"why":"Geometry-aware semantic correspondence supplies the feature extractor and similarity thresholds for filtering.","marker":"[53]"},{"why":"Kalman filtering is the stated inspiration for the probabilistic integration update.","marker":"[22]"}],"fun_headline_variants":["Probabilistic flow+keypoint fusion dominates TAP-Vid benchmarks","Fusing short-term flow with long-term matches wins tracking accuracy","ProTracker: Bayesian blend of flow and global cues sets tracking record","New tracker marries optical flow and heatmaps for precise point tracking","Uncertainty-aware tracker combines local and global signals for robust tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gain rests on treating the chained flow predictions as correlated Gaussians with a single constant correlation coefficient $p$, whose value is never reported, while also simplifying the flow Jacobian to orthogonal; if the true error correlations deviate from that model, the integration weights that produce the reported accuracy are not actually specified.","fun_headline_variants_meta":{"raw":{"variants":["Probabilistic flow+keypoint fusion dominates TAP-Vid benchmarks","Fusing short-term flow with long-term matches wins tracking accuracy","ProTracker: Bayesian blend of flow and global cues sets tracking record","New tracker marries optical flow and heatmaps for precise point tracking","Uncertainty-aware tracker combines local and global signals for robust tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3429,"prompt_tokens":849,"completion_tokens":2580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":2489}},"tokens_in":465,"tokens_out":2580,"duration_ms":18363,"temperature":1.0,"reasoning_tokens":2489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:53:03.054843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ProTracker on TAP-Vid DA VIS with $p$ fixed to 0, 0.5, and 1 and with the full flow Jacobian kept in Eq. (2); if $\\delta x_{\\text{avg}}$ does not peak near the paper's implicit $p$, or if the orthogonal-Jacobian version does not beat the full-Jacobian version, then the fusion rule as specified is not the mechanism behind the reported accuracy.","supporting_citations":[{"cited_title":"Mft: Long- term tracking of every pixel","cited_arxiv_id":null,"evidence_quote":"MFT provides the chaining schedule {∞,1,2,4,8,16,32} and the direct flow-chaining baseline that ProTracker extends."},{"cited_title":"Dino-tracker: Taming dino for self-supervised point tracking in a single video, 2024","cited_arxiv_id":null,"evidence_quote":"DINO-Tracker contributes the test-time-trained delta-DINO feature extractor and heatmap refiner used for long-term keypoints."},{"cited_title":"TAP-vid: A benchmark for track- ing any point in a video","cited_arxiv_id":null,"evidence_quote":"TAP-Vid defines the benchmark, the δx_avg/OA/AJ metrics, and the DA VIS and Kinetics evaluation protocols."},{"cited_title":"Tracking everything everywhere all at once","cited_arxiv_id":null,"evidence_quote":"Omnimotion supplies the Kinetics-100 subset and the optimization-based canonical-volume baseline for comparison."},{"cited_title":"Telling left from right: Identifying geometry-aware semantic corre- spondence","cited_arxiv_id":null,"evidence_quote":"Geometry-aware semantic correspondence supplies the feature extractor and similarity thresholds for filtering."},{"cited_title":"A new approach to linear filtering and prediction problems","cited_arxiv_id":null,"evidence_quote":"Kalman filtering is the stated inspiration for the probabilistic integration update."}],"review_version":1}