{"id":"38a61687-18b5-4cd5-8cc4-a6c0a1d5218a","arxiv_id":"2501.12375","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Video Depth Anything adapts Depth Anything V2 to produce temporally consistent depth for arbitrarily long videos using a temporal attention head, an optical-flow-free gradient loss, and key-frame-based stitching.","lead":"This paper presents a model that estimates depth for videos of several minutes with stable, flicker-free results while running in real time on a small model. It builds on the image depth model Depth Anything V2, adding a lightweight temporal processing head and a new training loss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal-consistency SOTA rests on TAE reported for only Scannet; TGM loss is sign-blind and masks dynamic-object motion, so the claimed 'arbitrarily long consistent depth' is under-supported.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test supports keeping that verdict rather than changing it. The sign-insensitivity of Eq. 3 is a real mathematical property, but its practical impact is partly mitigated by the per-frame SSI loss, so I do not elevate it to a fatal flaw. The more severe problem is evidentiary: the paper's headline claim about temporal consistency is backed by a single TAE column on one static indoor dataset, while dynamic and outdoor benchmarks are missing that metric. This is an absence of evidence for a core part of the claim, not an internal contradiction. The architecture and spatial results are credible; the efficiency numbers are independently plausible and the qualitative demonstrations are suggestive. However, to accept SOTA temporal consistency and 'arbitrarily long videos without compromising consistency,' the authors should report TAE across all evaluated datasets and across the frame-length sweep. My concrete test is designed to settle exactly that. If the missing measurements confirm the advantage, the conditional can be lifted; if not, the strong claim should be weakened.","tokens_in":16722,"tokens_out":3684,"duration_ms":40935,"concrete_test":"Compute TAE on all five zero-shot video benchmarks (KITTI, Scannet, Bonn, NYUv2, Sintel) under the same up-to-500-frame protocol used in Table 1 and Fig. 4, comparing VDA-L against DepthCrafter and DepthAnyVideo; report TAE at frame lengths 110, 192, 300, 400, and 500 for each dataset. Also compute a sign-sensitive temporal metric on Bonn, such as the fraction of pixels where the predicted depth-change direction disagrees with ground truth. If TAE on Bonn/Sintel or at longer frame lengths does not show a clear advantage, the 'new SOTA in temporal consistency' claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is simultaneous SOTA in spatial accuracy, temporal consistency, and efficiency for arbitrarily long videos. The load-bearing support is Table 1 and Fig. 4, but two gaps weaken it. (i) TAE, the only temporal-consistency metric, is reported only for Scannet (170 frames) in Table 1; it is not reported for KITTI, Bonn, NYUv2, or Sintel. The text claim that the model 'outperforms all baselines on all datasets in terms of temporal consistency' is therefore unsupported by the tables. Bonn and Sintel, containing dynamic scenes and large frame-to-frame depth changes, are exactly the cases where temporal consistency must be checked. (ii) The TGM loss in Eq. 3 compares absolute changes, ||d_{i+1}-d_i| - |g_{i+1}-g_i||_1, and is masked by |g_{i+1}-g_i| < 0.05. This discards the sign of depth change and excludes dynamic-object and edge regions. The single-image SSI loss anchors within-frame structure, but it does not force the correct direction of temporal depth change, nor does it constrain dynamic-object depth. In Fig. 4, only geometric metrics are shown as frame length increases; no TAE curve supports the claim that consistency is maintained 'without compromising.' The longest quantitative evaluation is 500 frames, so 'arbitrarily long videos' is also an extrapolation, though the keyframe mechanism is plausible. The argument would be materially stronger with TAE on all datasets and at all evaluated lengths.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Video Depth Anything, a feed-forward video depth estimator built on the Depth Anything V2 image model. It replaces the DPT head with a spatio-temporal head that includes temporal self-attention layers, trains with a newly proposed temporal gradient matching (TGM) loss together with the scale-shift-invariant image loss, and develops a key-frame-based segment-wise inference strategy for long videos. The authors claim state-of-the-art zero-shot video depth estimation on five benchmarks in terms of geometric accuracy, temporal consistency, and inference efficiency, and they further claim that the model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization. The paper provides quantitative results for geometric accuracy on KITTI, Scannet, Bonn, NYUv2, and Sintel; temporal consistency (TAE) results are reported only for a 170-frame Scannet subset in the main table. Ablations cover loss functions, inference strategies, window sizes, and image distillation.","tokens_in":17089,"tokens_out":6095,"duration_ms":52006,"significance":"If the result holds, this is a meaningful advance: it shows that a lightweight transformer with a small temporal head can match or beat diffusion-based video depth models on geometric accuracy while being substantially faster, and that long-video inference can be stabilized with a simple key-frame reference strategy. The empirical evaluation is extensive on the geometric side, the ablations are informative, and the latency numbers are compelling. However, the central temporal-consistency claim is under-supported because the only temporal metric, TAE, is reported for a single dataset, and the proposed TGM loss is sign-insensitive and masked on dynamic regions, so the training signal does not directly enforce correct temporal direction or dynamic-object consistency. These gaps need to be addressed before the SOTA claim can be fully accepted.","major_comments":[{"comment":"Table 1 reports TAE, the only temporal-consistency metric, exclusively for the Scannet 170-frame subset; no TAE values are given for KITTI, Bonn, NYUv2, or Sintel. The sentence in Sec. 4.2 that the model 'outperforms all baselines on all datasets in terms of temporal consistency' is therefore not supported by the reported numerical evidence. This is particularly important because Bonn is a dynamic indoor dataset and Sintel contains large motions, exactly the conditions where temporal consistency and dynamic-object behavior need to be verified. Please report TAE on all five datasets, or at least on Bonn and Sintel, and include TAE as a function of frame length in Fig. 4 if the 'without compromising' claim is to be sustained.","section":"Table 1, Sec. 4.2"},{"comment":"The TGM loss in Eq. (3) is LTGM = (1/(N-1)) sum || |d_{i+1}-d_i| - |g_{i+1}-g_i| ||_1, computed only where |g_{i+1}-g_i| < 0.05. This formulation is invariant to the sign of the temporal depth change: a prediction that moves in the opposite direction from the ground truth with the same magnitude incurs no loss. Moreover, the threshold explicitly excludes dynamic-object and edge regions, so the model receives no direct supervision for the temporal consistency of moving objects. Since the paper's main temporal-consistency claim is credited to this loss, the sign-blindness and the mask are load-bearing. The ablation in Table 4 does not include a sign-aware or dynamic-region-aware variant, so the contribution of these design choices is untested. I recommend ablating a sign-sensitive version of the loss (e.g., enforcing signed gradient similarity) and reporting a directional temporal metric.","section":"Eq. (3), Sec. 3.2"},{"comment":"The claim that the model can be applied to 'arbitrarily long videos without compromising quality, consistency, or generalization ability' is an extrapolation from the presented evidence. The longest quantitative evaluation is 500 frames (Fig. 4), and the only longer demonstration is the qualitative 7320-frame self-captured video in Fig. 7, where no quantitative consistency metric is reported. While the key-frame mechanism is plausible, the paper should either temper the 'arbitrarily long' phrasing or provide quantitative results at lengths beyond 500 frames, e.g., on the self-captured video or by reporting TAE every N frames for a longer sequence.","section":"Sec. 4.2, Fig. 4, Fig. 7"}],"minor_comments":[{"comment":"The word 'wrapping' should be 'warping', and the summation index i starts at 2; please clarify whether the index range is intended to be i = 1 to N-1.","section":"Sec. 3.2, Eq. (1)"},{"comment":"'ChoronDepth' is a typo for 'ChronoDepth' in the table header and row labels.","section":"Table 1"},{"comment":"'Througout' should be 'Throughout'.","section":"Sec. 4.3"},{"comment":"The definition of consistency as 'the maximum TAE among all models minus the TAE of each individual model' is confusing as written; please define it clearly in the caption or text.","section":"Fig. 1 (right)"},{"comment":"'with a overlap' should be 'with an overlap' in the window-size discussion.","section":"Supplementary Sec. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core method is promising enough to merit a major revision. The main issues are the thin temporal-consistency evidence and the sign-blindness of the TGM loss; both are addressable with additional experiments. I would not reject on these grounds, but the current claims exceed the reported evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you'll want to know about is Video Depth Anything, a feed-forward video depth model built on a frozen Depth Anything V2 encoder with a lightweight temporal attention head and a new loss that avoids optical flow. The headline claim is state-of-the-art spatial accuracy, temporal consistency, and efficiency for arbitrarily long videos.\n\nWhat's genuinely new and good: the architecture is simple and effective, the efficiency figures are strong (VDA-L at 67 ms/frame, VDA-S at 9.1 ms/frame on A100), and the geometric accuracy is evaluated across five datasets and beats prior feed-forward methods by wide margins. The key-frame referencing inference strategy is a plausible answer to accumulated scale drift, and the ablations show it helps. The paper also preserves DAv2's zero-shot image performance, which is not trivial.\n\nThe weak spot is exactly where the title puts emphasis: temporal consistency. The only temporal metric in Table 1 is TAE, and it's reported for just one dataset (Scannet at 170 frames). The text claims to outperform all baselines on all datasets in terms of temporal consistency, but that claim is unsupported by the tables; the supplementary short-video table also lacks TAE. The TGM loss in Eq. 3 is sign-insensitive: it compares absolute changes, so a predicted depth change in the wrong direction incurs no penalty. The threshold |g_{i+1}-g_i| < 0.05 masks dynamic regions entirely, so dynamic-object depth is unconstrained. Table 4 shows TGM+SSI is slightly worse than SE+SSI on TAE, undercutting the claim that TGM is superior to the optical-flow-based stable error. The 'arbitrarily long' claim is extrapolated from 500-frame evaluations, though the qualitative minutes-long demo suggests it has legs. The limitation section honestly notes the small training data, but the temporal-consistency evidence is thinner than the writing suggests.\n\nWho's this for? Anyone working on video depth or temporal consistency in monocular estimation will find the engineering results useful, and the loss and inference ideas are worth discussing. The paper deserves a serious referee: the core idea is sound, and the flaws are in the evidence and loss formulation, not the architecture. I'd send it to review, asking for TAE across all datasets and a direct discussion of the sign issue.","headline":"Efficient video depth with a strong architecture and wide geometric experiments, but the temporal-consistency flagship claim rests on a single TAE column and a sign-blind loss; still worth a serious referee.","tokens_in":17599,"tokens_out":4924,"would_cite":true,"duration_ms":39243,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video Depth Anything claims that one feedforward model can produce temporally consistent depth for arbitrarily long videos while retaining the spatial accuracy of its monocular base model, without optical flow, camera poses, or…","keywords":["monocular depth estimation","video depth estimation","temporal consistency","temporal gradient matching","spatiotemporal attention","long video inference","zero-shot generalization","Depth Anything V2"],"falsifier":"On a static-camera clip of an object moving toward the camera, check whether the model's depth at the object's pixels decreases over time; if it instead increases with the same absolute magnitude, TGM training has no loss term to correct it. A signed depth-change error computed on such dynamic clips would settle whether the consistency is genuine.","tokens_in":16526,"feed_emoji":"🎥","tokens_out":4853,"duration_ms":44657,"temperature":0.7,"pith_summary":"Video Depth Anything sets out to solve a practical bottleneck in monocular depth estimation: applying a strong image-depth model to video produces flicker, while existing video-depth methods only handle clips under ten seconds and trade quality for compute. The paper claims a single feedforward model, built on Depth Anything V2, can estimate temporally consistent depth for videos lasting several minutes with no drop in spatial accuracy or generalization. It achieves this with a spatiotemporal head inserted into the depth head, a temporal gradient matching loss that needs no optical flow or pose supervision, and a key-frame-based inference strategy for arbitrarily long videos. If these claims hold, the model would give robotics, AR, and video-editing pipelines a fast and consistent depth source for long footage.","feed_headline":"Depth estimates stay consistent over minute-long videos at 30 fps","feed_subtitle":"Matches diffusion-based methods in accuracy while running in real time, with no optical-flow or pose priors.","key_machinery":"The load-bearing mechanism is the Temporal Gradient Matching (TGM) loss: at each pixel coordinate, the absolute change in predicted depth between adjacent frames is forced to match the absolute change in ground-truth depth, computed only where the ground-truth change is below 0.05 to avoid edges and dynamic objects. Because the loss uses the same coordinate rather than optical-flow correspondences, it needs no geometric priors. It is supported by a spatiotemporal head that applies temporal self-attention inside the DPT head, and by an inference strategy that feeds each new window together with overlapping frames and two key frames spaced 12 frames apart, then linearly interpolates predictions in the overlap region to avoid flicker and scale drift.","core_discovery":"The paper's central claim is that temporal consistency in video depth estimation can be obtained by training a spatial-temporal head on top of a frozen Depth Anything V2 encoder, using a simple loss that matches the absolute temporal gradient of predicted depth to that of ground truth, and then stitching inference windows with overlapping frames plus key frames from earlier clips. On five zero-shot video benchmarks the model reports state-of-the-art spatial accuracy on four datasets and best temporal consistency on all five, while retaining image-depth accuracy close to Depth Anything V2; the small variant runs at 30 FPS. The authors attribute this to the combination of a lightweight temporal attention head, the temporal gradient matching loss, joint training on video depth plus unlabeled single images, and the window-stitching scheme that inherits scale and shift information across windows.","pith_inferences":["The sign-agnostic form of TGM means a prediction that reverses the direction of depth change could match the loss as long as the magnitude is right; a benchmark measuring signed change on dynamic objects would reveal whether this matters.","The key-frame interval and overlap size are fixed hyperparameters; very fast camera motion or rapid scene changes could decorrelate the key frames from the current window, so adaptive spacing is a natural extension.","Because the encoder is frozen and only the head sees video data, the temporal reasoning capacity is limited; fine-tuning the backbone with more video data could improve dynamic-object consistency beyond the current results.","The reported efficiency advantage suggests depth from video could be used as a real-time input modality for downstream tasks such as point-cloud fusion, rather than as an offline post-processing step."],"forward_implications":["Videos of several minutes can be processed by a single forward pass per window without cumulative drift, removing the practical ceiling that limited earlier methods to clips under about ten seconds.","The smallest model's 30 FPS runtime makes temporally consistent depth a real-time option for robotics, AR, and 3D video conversion.","Zero-shot image depth accuracy is preserved almost unchanged, so one model can serve both still-image and video tasks.","Because no optical flow, camera-pose, or video-diffusion priors are used, the approach avoids the failure modes and compute cost those priors introduce."],"supporting_citations":[{"why":"Supplies the frozen Depth Anything V2 encoder and teacher for image distillation that the method extends.","marker":"[43]"},{"why":"Provides the DPT reassemble and fusion head architecture into which temporal layers are inserted.","marker":"[28]"},{"why":"Provides the scale-shift-invariant spatial loss and multi-dataset training recipe reused in the total loss.","marker":"[3]"},{"why":"Defines the optical-flow-based OPW loss that the TGM loss is designed to replace and serves as a baseline.","marker":"[39]"},{"why":"Primary video-diffusion baseline and the source of the 110-frame evaluation protocol used for short-video comparisons.","marker":"[13]"},{"why":"Baseline for long and short video depth and the source of the Temporal Alignment Error metric used to measure consistency.","marker":"[41]"}],"fun_headline_variants":["Consistent depth for super-long videos at 30 FPS","Real-time video depth without optical flow priors","Zero-shot video depth sets new state of the art","Minute-long videos get flicker-free depth estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training loss assumes that matching the absolute size of depth change at each pixel between adjacent frames, and only where ground-truth change is tiny, is enough to make depth temporally consistent, so wrong-direction changes and dynamic-object motion are never directly penalized.","fun_headline_variants_meta":{"raw":{"variants":["Consistent depth for super-long videos at 30 FPS","Real-time video depth without optical flow priors","Zero-shot video depth sets new state of the art","Minute-long videos get flicker-free depth estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2527,"prompt_tokens":938,"completion_tokens":1589,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1525}},"tokens_in":554,"tokens_out":1589,"duration_ms":12248,"temperature":1.0,"reasoning_tokens":1525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:13:21.212957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a static-camera clip of an object moving toward the camera, check whether the model's depth at the object's pixels decreases over time; if it instead increases with the same absolute magnitude, TGM training has no loss term to correct it. A signed depth-change error computed on such dynamic clips would settle whether the consistency is genuine.","supporting_citations":[{"cited_title":"Vi- sion transformers for dense prediction","cited_arxiv_id":null,"evidence_quote":"Provides the DPT reassemble and fusion head architecture into which temporal layers are inserted."}],"review_version":1}