{"id":"ce661234-0331-42a1-b9a1-eeb5867cf3de","arxiv_id":"2411.17249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"By adding temporal attention and an optical-flow-based stability loss to a frozen single-image depth/normal model, Buffer Anytime obtains temporally consistent video geometry without any paired video supervision.","lead":"Buffer Anytime trains video depth and normal estimators without paired video ground truth, combining a frozen single-image model with an optical-flow consistency loss and lightweight temporal attention. If the results hold, video geometry estimation can borrow the accuracy of image models and avoid expensive annotated video datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stabilization loss enforces warp-invariance of depth/normal, which is geometrically invalid under camera motion; OPW gains may be artifacts of optimizing an incorrect objective.","rationale":"The reader's weakest_assumption is the same as the concern that I find most load-bearing: the optical-flow stabilization loss assumes warp-invariance of depth and normal maps, which is not the correct physical relationship under camera motion. The paper's response to flow inaccuracy (cycle validation and edge masking) does not cure this: those masks remove outliers and boundaries, not the systematic viewpoint-change inconsistency that affects ordinary static pixels during ego-motion. The circularity with the OPW metric strengthens the concern: the paper reports that it improves the same quantity it trains to minimize, and OPW is described as 'defined similarly to our smoothness loss.' I agree with the reader's assessment that this creates a real risk that the temporal-consistency improvements are smoothing artifacts. I do not, however, think the evidence forces a REJECT: the per-frame metrics (AbsRel, δ1) still improve over the image backbone, suggesting the regularization loss anchors the model to the image prior, and the method may still produce useful temporal coherence in practice even if the loss is not physically exact. The conditions the reader attaches—release of code/data, normals baselines, geometric analysis of the warp loss—are appropriate. Therefore I leave the verdict as CONDITIONAL (UNCHANGED). A single concrete check, comparing predicted depth ratios at flow-corresponded pixels against ground-truth ratios in a forward-motion sequence, would directly settle whether the OPW improvements reflect real geometry or an artifact of an invalid loss.","tokens_in":13803,"tokens_out":6371,"duration_ms":63412,"concrete_test":"On KITTI (or any sequence with ground-truth depth and known ego-motion or poses), compute the ground-truth depth ratio r_gt(x) = D_gt_{k+1}(x') / D_gt_k(x) for corresponding pixels x' = O_{k→k+1}(x), focusing on pixels with significant flow and depth change (e.g., forward motion). Compute the same ratio r_pred for the predicted depth maps from Buffer Anytime. If the method has learned true geometric consistency, r_pred should reproduce the distribution of r_gt, including values far from 1. If instead r_pred is systematically compressed toward 1 and the error |log r_pred − log r_gt| increases with |log r_gt|, then L_stable has trained the model toward warp-invariance rather than geometric correctness. This test distinguishes genuine video-geometry understanding from smoothing toward an invalid target, settling whether the OPW gains are meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim hinges on the optical-flow stabilization loss L_stable (Eqs. 4–5, Algorithm 2), which encourages the predicted depth/normal at a pixel in frame k to equal the predicted value at the flow-corresponded pixel in frame k+1. For a static scene under camera motion this identity is geometrically false: with translation t and rotation R, a 3D point has depth D_{k+1}(x') = (R D_k(x) K^{-1} x + t)_z, which generally differs from D_k(x); camera-coordinate normals transform as N_{k+1}(x') = R N_k(x), not N_k(x). The cycle-validation and Canny-edge masks (Sec. 3.2) only remove flow outliers and depth discontinuities; they do not model viewpoint change. Because the evaluation metric OPW is 'defined similarly to our smoothness loss' (Sec. 4.1), minimizing L_stable directly lowers OPW even when the model converges to a warp-invariant but geometrically incorrect solution—e.g., suppressing the depth change caused by forward ego-motion. The reported AbsRel/δ1 improvements are necessary but not sufficient for the claim, since they do not establish that the learned temporal consistency reflects correct 3D geometry. Without a geometric check, the zero-shot claim remains vulnerable: the OPW gains could be an artifact of fitting an invalid objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Buffer Anytime, a training framework that converts a single-image depth or normal estimator into a video estimator without paired video-geometry ground truth. The method fine-tunes an image backbone (Depth Anything V2 for depth, Marigold-E2E-FT for normals) with lightweight temporal attention layers, using a hybrid loss: a per-frame regularization loss that keeps predictions aligned with the frozen image model, and an optical-flow-based stabilization loss that penalizes differences between neighboring frames at flow-corresponded pixels. Training is performed on roughly 200K unlabeled videos. Experiments on ScanNet, KITTI, Bonn, and Sintel report improved temporal consistency (OPW) and maintained or slightly improved per-frame accuracy relative to the image baselines, with depth results claimed to be comparable to video-trained methods such as DepthCrafter.","tokens_in":14039,"tokens_out":3888,"duration_ms":38325,"significance":"If the main claim is validated, the framework is practically significant: it offers a recipe for upgrading strong single-image geometry models to temporally coherent video predictors while bypassing expensive paired video-depth and video-normal datasets. The paper is careful in several respects: it evaluates on multiple standard benchmarks, reports ablations of regularization weight and masking, provides pseudocode for the deferred back-propagation and stabilization losses, and explicitly acknowledges limitations of the backbone and of adjacent-frame-only flow guidance. The core idea of combining image priors with optical-flow stabilization is appealing and likely to be influential. The central unaddressed risk is that the temporal-consistency objective is not geometrically correct under camera motion; because the evaluation metric OPW is defined similarly to the training loss, the reported consistency gains may partly reflect optimization of a non-geometric target.","major_comments":[{"comment":"The ablation studies do not isolate the contribution of the temporal attention module from the contribution of the stabilization loss. In Supp. Table 4, \"Ours w/o fine-tuning\" (temporal layers trained with the frozen backbone refinement network) already gives 0.121 AbsRel and 0.040 OPW, close to the full model's 0.119 and 0.038, and \"Ours all frames\" gives 0.120/0.865/0.035, slightly better OPW than the full model's 0.038. The manuscript would be strengthened by a variant that uses the same stabilization loss on the image backbone without any temporal attention, and a variant with temporal attention but without L_stable, so the reader can see which component drives the reported gains.","section":"Sec. 4.3 and Supp. Table 4"}],"minor_comments":[{"comment":"The main text says the training process takes approximately one day for 20,000 iterations, while the supplementary says approximately one day for 15,000 iterations. These numbers should be reconciled.","section":"Supp. Sec. 7"}],"recommendation":"major_revision","confidential_remarks":"The geometric validity of the stabilization loss is the central concern, and I believe it is addressable: the authors could add a geometric-consistency evaluation, report a metric not aligned with the training loss, and either justify or revise the warp-invariance assumption. The paper does not currently release code or data, which weakens reproducibility given that the training set (200K videos) is not described in detail. The fit with the journal's scope is good, but the verifiability of the main claim will depend on the additional geometric evidence requested above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2411.17249. First, it ships a real, working recipe: freeze a strong single-image depth/normal model, add lightweight temporal attention blocks, and fine-tune with a hybrid loss that combines an image-prior regularization term with an optical-flow warp loss. No paired video-geometry data is used. On the DepthCrafter benchmark it beats Depth Anything V2 on AbsRel/δ1 on ScanNet and KITTI, and it lands close to DepthCrafter itself. That is a solid empirical result, and the normal extension is a nice extra. Second, the headline temporal-consistency metric, OPW, is defined almost identically to the stabilization loss itself, so OPW gains are partly a check that optimization worked, not independent evidence that the geometry is correct. The AbsRel/δ1 numbers are independent and do support the practical claim.\n\nWhat is genuinely new is the combination: single-image priors + optical-flow stability + temporal attention for zero-shot video buffers. Each ingredient is known, but the package is new and works. The paper is honest about its main limitation in the Discussion: the flow loss only links adjacent frames and can miss objects that leave and return. It also explicitly floats \"losses defined in 3D space\" as future work.\n\nThe geometric worry you raised is real but in proportion. The warp loss enforces that a pixel and its flow-corresponded neighbor have the same depth/normal value. For a static point under camera translation, depth changes; under rotation, camera-coordinate normals change. So the loss is not a correct rigidity constraint. The cycle-validation and Canny masks filter outliers and depth edges but do not model viewpoint. However, in practice the loss acts as a soft temporal smoother on top of a strong image prior, and the independent accuracy metrics do not collapse. The bigger soft spot is that the training data is underspecified—\"approximately 200K videos\" with no source or domain description—and the normal evaluation has no video-model baseline, only single-image methods. That makes the bold \"comparable to video models\" claim well-supported for depth but unexamined for normals.\n\nThe citation pattern looks straightforward. The paper builds on NVDS, ChronoDepth, DepthCrafter, and AnimateDiff, and its related work is fair. No code or data is released in the paper, which weakens reproducibility but does not undermine the reported numbers.\n\nWho is this for? Anyone working on video depth or normal estimation, or on turning image priors into video predictors. It deserves a serious referee: the method is plausible, the experiments are nontrivial, and the geometric concern is a legitimate review point rather than a reason to reject. I would engage with it and ask for code/data release, a clearer dataset description, and an analysis of what the warp loss actually enforces.","headline":"A genuinely useful zero-shot recipe for video depth/normal that beats its image backbones and approaches video-supervised models, but the temporal-consistency metric mirrors the training loss and the warp loss enforces an oversimplified geometry.","tokens_in":14627,"tokens_out":2549,"would_cite":true,"duration_ms":23529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video depth and normal maps can be learned from unlabeled video by keeping a single-image model frozen and adding an optical-flow smoothness loss.","keywords":["video depth estimation","surface normal estimation","geometric buffers","zero-shot training","optical flow","temporal consistency","single-image priors","temporal attention"],"falsifier":"A decisive test is to render a synthetic video of a static scene with known ground-truth depth and normals while the camera translates sideways and rotates, then check whether the model's predictions at flow-corresponding pixels are biased toward equal values by an amount that grows with inter-frame motion and object depth.","tokens_in":13565,"feed_emoji":"🎬","tokens_out":8258,"duration_ms":72999,"temperature":0.7,"pith_summary":"This paper sets out to show that temporally consistent video depth and surface-normal maps, which the authors call geometric buffers, can be trained without any paired video-geometry supervision. The recipe freezes a strong single-image estimator, inserts lightweight temporal attention blocks, and fine-tunes only those blocks under a hybrid loss: a regularization term keeps each frame close to the image model's prediction, and an optical-flow-based term asks that pixels matched between adjacent frames receive the same predicted value. Applied to Depth Anything V2 for depth and Marigold-E2E-FT for normals, the method improves temporal consistency over the image backbones while keeping per-frame accuracy, and its depth results are comparable to a state-of-the-art video model trained on large-scale paired video data. The practical stake is that expensive video-depth and video-normal annotation may be replaceable by unlabeled video plus optical flow.","feed_headline":"Unlabeled video can train consistent depth and normals","feed_subtitle":"A frozen single-image model plus optical-flow smoothing matches video models trained on paired ground truth.","key_machinery":"The load-bearing object is the hybrid loss, L = ω_reg L_reg + L_stable, computed while the video model processes K frames. L_reg is an affine-invariant relative depth loss (or an L2 loss on latent normal maps) against the fixed image model, which anchors each frame to the image prior; L_stable warps the predicted buffer from one frame to the next using a pretrained optical-flow model and penalizes L1 differences at flow-corresponding pixels, in both forward and backward directions. Because optical flow is imperfect, only pixels that pass cycle validation and are not near Canny edges of the predicted buffer contribute to L_stable. The temporal architecture injects zero-initialized temporal attention blocks between frozen layers of the backbone, so training starts from the image model and adds cross-frame reasoning; for the normal model, deferred back-propagation keeps the stabilization loss computable by decoding chunks of latent maps.","core_discovery":"The central claim is that the temporal inconsistency of image-based geometry models is not a defect that requires video ground truth to fix. A video model can be produced by taking an image prior, freezing most of its weights, inserting temporal attention layers that start from zero, and training with a loss that both preserves the image model's per-frame predictions and pulls flow-corresponding pixels toward equal depth or normal values. On the DepthCrafter benchmark, this zero-shot model improves on Depth Anything V2 on AbsRel, δ1, and OPW across ScanNet, KITTI, and Bonn, and reaches accuracy and smoothness comparable to video-supervised models such as DepthCrafter despite seeing no paired video-depth data. For normals, per-frame angular errors stay close to Marigold-E2E-FT while temporal smoothness improves sharply, e.g., OPW drops from 0.152 to 0.065 on Sintel and 0.092 to 0.069 on ScanNet.","pith_inferences":["Beyond the paper: the same skeleton should transfer to other buffer types the paper lists but does not test, such as lighting or material maps, as long as a strong single-image estimator exists and the regularization term is adapted to that buffer.","Beyond the paper: a 3D-aware variant of the stabilization loss, one that warps using estimated camera motion or rigid scene flow instead of raw optical flow, would isolate how much of the remaining temporal error comes from the warp-invariance approximation.","Beyond the paper: because only RGB video and optical flow are needed, collecting more unlabeled video should translate into better temporal consistency at no annotation cost, opening a scaling route that paired-data video models do not have."],"forward_implications":["Temporal consistency of video depth improves without sacrificing accuracy: on KITTI, OPW drops from 0.089 for Depth Anything V2 to 0.038 for the fine-tuned video model, with AbsRel also improving.","Zero-shot training reaches parity with paired-data video methods: on ScanNet and KITTI, the depth model is comparable to DepthCrafter in accuracy and better in OPW, despite never seeing video-depth ground truth.","Video normal estimation becomes a self-supervised problem: per-frame angular accuracy stays near the Marigold-E2E-FT backbone while temporal smoothness improves on both Sintel and ScanNet.","Training geometry-video models is cheap enough to scale: roughly one day on 24 H100 GPUs, because the backbone is frozen and only temporal layers plus a light fine-tuning head are trained."],"supporting_citations":[{"why":"Supplies the frozen image-depth backbone (Depth Anything V2) that the video depth model fine-tunes around.","marker":"[56]"},{"why":"Supplies the image-normal backbone (Marigold-E2E-FT) whose latent predictions anchor the normal model and define the regularization loss.","marker":"[20]"},{"why":"Provides the temporal attention block design (AnimateDiff) inserted between spatial layers to add cross-frame reasoning.","marker":"[24]"},{"why":"Serves as the main video-depth baseline and supplies the benchmark, test splits, and OPW metric used to show parity without paired video data.","marker":"[30]"},{"why":"Introduces the optical-flow-based stabilization idea for video depth that L_stable builds on.","marker":"[53]"},{"why":"Supplies deferred back-propagation, which lets the normal model compute the stabilization loss on decoded chunks within memory limits.","marker":"[60]"}],"fun_headline_variants":["Video depth without video labels? Just add flow","Zero-shot video depth from single-image priors","Image priors + flow = video consistency","Video geometry from image priors alone","Frozen image model + flow gives video-depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two pixels linked by optical flow between adjacent frames should be assigned the same depth value or the same normal vector, which is exactly true only for a static scene seen without rotation; under camera translation the true depth at corresponding pixels differs, and under rotation the true normal direction changes, so the stabilization target is an approximation rather than a geometric ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Video depth without video labels? Just add flow","Zero-shot video depth from single-image priors","Image priors + flow = video consistency","Video geometry from image priors alone","Frozen image model + flow gives video-depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":2997,"prompt_tokens":882,"completion_tokens":2115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2047}},"tokens_in":498,"tokens_out":2115,"duration_ms":14599,"temperature":1.0,"reasoning_tokens":2047,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:21:32.202459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to render a synthetic video of a static scene with known ground-truth depth and normals while the camera translates sideways and rotates, then check whether the model's predictions at flow-corresponding pixels are biased toward equal values by an amount that grows with inter-frame motion and object depth.","supporting_citations":[{"cited_title":"Arf: Artistic radiance fields","cited_arxiv_id":null,"evidence_quote":"Supplies deferred back-propagation, which lets the normal model compute the stabilization loss on decoded chunks within memory limits."}],"review_version":1}