Pith. sign in

REVIEW 3 major objections 4 minor 88 references

Video Depth without Video Models

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RollingDepth shows that a single-image latent diffusion depth model, applied to short overlapping snippets and globally co-aligned, produces video depth more accurate than dedicated video depth estimators and high-performing single-frame…

desk verdict A strong empirical paper that shows an image LDM with snippet stitching beats video diffusion models for video depth; the main caveats are missing code, missing variance, and an untested affine-consistency assumption. read the letter →

arxiv 2411.19189 v2 pith:X3M6HZK3 submitted 2024-11-28 cs.CV

Bingxin Ke , Dominik Narnhofer , Shengyu Huang , Lei Ke , Torben Peters , Katerina Fragkiadaki , Anton Obukhov , Konrad Schindler This is my paper · ORCID
classification cs.CV
keywords videodepthestimationlatentdiffusionmodelmonoculartemporalconsistencyglobalco-alignmentcross-frameself-attentioninversezero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a single-image latent diffusion depth model can become a strong video depth estimator without training a video diffusion model. Its recipe is to process a video as many overlapping three-frame snippets, sampled at different frame spacings, with the depth model's attention mechanism operating jointly across each snippet's frames; each snippet yields its own inverse-depth prediction. A robust optimization then fits one scale and one shift per snippet so that all overlapping predictions agree, and this global co-alignment is what mainly produces the accuracy gain. The resulting system, RollingDepth, is reported to be more accurate than both dedicated video depth estimators and strong single-frame models on PointOdyssey, ScanNet, Bonn, DyDToF, and DDAD, while handling videos of hundreds of frames. If the claim holds, accurate video depth no longer requires the expensive training and stitching routines of video foundation models.

What carries the argument

The central machinery is the pair formed by the dilated rolling snippet kernel and the per-snippet scale-and-shift co-alignment. The kernel selects frame triplets $(x_{i-g}, x_i, x_{i+g})$ for multiple dilation rates $g$, so every frame participates in many snippets with different temporal contexts; cross-frame self-attention inside the latent diffusion model lets depth evidence flow across the three frames of a snippet. The co-alignment step then assembles all snippets into one video by minimizing a robust L1 objective over the mean depth and mean inverse depth per frame, fitting one pair $(s_k, t_k)$ per snippet with soft constraints and stability weights (Eq. 3). Because inverse depth is normalized jointly within each snippet during training (Sec. 3.4), the same frame is represented comparably even when it appears in different contexts, which is what allows a single global scale-and-shift per snippet to register everything.

What would settle it

Take a fixed frame and feed it to the snippet model in two different contexts—as the middle frame of an adjacent triplet (frames 99, 100, 101) and of a dilated triplet (frames 90, 100, 110). After optimally aligning each snippet's output with its own scale and shift, compare the two depth maps of frame 100; because per-snippet normalization renormalizes the same frame differently by context, a residual difference larger than the method's reported error would show that the single-scale-and-shift assumption fails and the global alignment cannot repair it.

Watch

Extended reading notes

Core claim

The central discovery is that temporal consistency in video depth can be obtained from a single-image latent diffusion model by splitting the video into short snippets and solving a global alignment problem, rather than by learning a video prior. The multi-frame snippet model is fine-tuned from a single-image depth LDM by letting self-attention attend across the frames of a snippet; per-snippet inverse-depth normalization keeps depth ranges comparable within a snippet and lets the model handle sudden changes in depth range. Snippets are drawn with dilation rates $g \in \{1,10,25\}$, so each frame appears with near and far temporal neighbors. The co-alignment then optimizes one scale $s_k$ and shift $t_k$ per snippet, minimizing an L1 discrepancy over all overlapping frames under soft constraints (Eq. 3). Ablations show that the co-alignment step is the largest contributor to the accuracy gain, while the optional diffusion refinement mostly improves visible detail.

Load-bearing premise

The pipeline assumes that within any three-frame snippet the depth estimates are already mutually consistent up to one shared multiplication and addition, so the alignment needs only one scale and shift per snippet rather than per frame.

Editorial extensions

If this is right

  • A 250-frame video can be processed in roughly 81 to 105 seconds on one GPU, with a constant memory footprint, because the network only ever sees short snippets; this makes the method applicable to long videos of hundreds of frames.
  • Dilation matters: adding a large frame spacing (25) to the base spacing (1) reduces the relative depth error by more than six percentage points on PointOdyssey and more than two on ScanNet, and an intermediate spacing (10) adds a further small gain.
  • Co-alignment is the decisive component: ablations show that merging snippets without it gives about 13.0 relative error on PointOdyssey, versus 10.2 with it, while the optional refinement changes the metric only marginally.
  • Across all reported benchmarks, RollingDepth achieves lower relative depth error and higher accuracy ($\delta_1$) than the compared dedicated video depth estimators and single-frame models, including on the long-sequence subsets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence of this design is that the global co-alignment could be reused with any per-snippet dense predictor that outputs affine-invariant maps, not only diffusion-based depth; if alignment is doing most of the work, a cheaper frame-wise predictor might close much of the gap on static scenes.
  • The method reframes video depth as largely a registration problem for per-snippet depth ranges, rather than a motion-generation problem; comparing RollingDepth against a flow-based or optimization-based registration baseline would isolate how much the cross-frame attention itself contributes.
  • The optional refinement, which adds moderate noise and denoises with decreasing frame spacings, acts as a learned detail enhancer; measuring its effect on long-range flicker (not just per-frame metrics) would show whether it also improves temporal consistency.
  • Because the same frame is normalized differently depending on the snippet it appears in, the approach may be sensitive to snippet sampling; stress-testing with extreme frame spacings or rapid scene cuts could reveal whether intra-snippet consistency, rather than global registration, becomes the bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RollingDepth, a video depth estimation method that extends the single-image latent diffusion model Marigold to short video snippets by retraining it with cross-frame self-attention and per-snippet inverse-depth normalization. At inference, snippets of three frames are sampled with a dilated rolling kernel at multiple dilation rates, processed by the snippet LDM, and merged into a consistent depth video by a global optimization that estimates one scale and shift per snippet, followed by averaging; an optional diffusion-based refinement step improves detail. The method is trained on TartanAir and Hypersim and evaluated zero-shot on PointOdyssey, ScanNet, Bonn, DyDToF, and DDAD, where it reports lower AbsRel and higher δ1 than single-frame and video-based baselines on most benchmarks, with additional temporal smoothness metrics in the supplementary.

Significance. If the central claims hold, the paper makes a valuable practical contribution: it shows that a single-image depth diffusion model can be turned into a state-of-the-art video depth estimator with modest additional training, avoiding the high cost of video diffusion models. The multi-scale snippet sampling and robust global co-alignment are clean, well-ablated ideas, and the evaluations cover several standard benchmarks with standard protocols. The paper also acknowledges several limitations (sky, glass, occasional failures), which strengthens credibility. However, the significance is tempered by the overbroad headline claim and the untested affine-consistency assumption that underlies the co-alignment step, both of which require attention before the paper can be accepted.

major comments (3)
  1. [Sec. 3.3, Eq. (3)] The global co-alignment assumes each snippet's depth output is related to a common scale by a single affine transform per snippet. This assumption is load-bearing because Table 3 attributes most of the accuracy gain to co-alignment (PointOdyssey AbsRel 13.0 without vs 10.2 with; ScanNet 12.4 vs 9.9). The paper never directly measures intra-snippet consistency; if the multi-frame LDM produces non-affine errors (e.g., errors that vary nonlinearly with depth or depend on the temporal context), the co-alignment cannot correct them and the fused video inherits them. I recommend adding an experiment that fits a single scale/shift per snippet to ground truth and reports the residual intra-snippet error, or that compares predictions of overlapping snippets before and after alignment to quantify the fraction of inconsistency that is affine-correctable.
  2. [Abstract, Sec. 4.3, Table 1] The claim that RollingDepth 'delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models' is contradicted by the Bonn row of Table 1: DepthCrafter achieves AbsRel 6.6 and δ1 96.7 versus RollingDepth's 7.9 and 93.9. The conclusion's 'best-in-class performance across multiple datasets' is likewise too strong. The paper should qualify the claim, e.g., 'on average across the evaluated benchmarks' or 'on all but the Bonn benchmark,' and discuss why the method underperforms there.
  3. [Sec. 4.2, Table 1, Supplementary A.3] The evaluations report point estimates without error bars or significance tests, and the PointOdyssey test set is filtered by removing toy scenes, smoke, and window pixels; the exact subset is promised only as 'will be provided with the code.' Since the PointOdyssey results are among the strongest claims, the paper should provide the selected sequence list, report per-sequence variance (e.g., interquartile ranges), and verify that the baselines are evaluated on the same filtered subset. Without this, a reader cannot assess whether the gaps are within noise or a consequence of the filter.
minor comments (4)
  1. [Supplementary Tab. S1] The OPW table marks RollingDepth with an asterisk for catastrophic failures on PointOdyssey, but the text only discusses failures for ChronoDepth and DepthCrafter. Please clarify whether RollingDepth also has such failures and how they were handled.
  2. [Sec. 4.4 vs Table S4] The main text states the fast setting takes 81 s, while Table S4 reports 105 s for the standard configuration and 81 s for the batched variant; please reconcile the numbers and specify which configuration corresponds to the 'fast' setting in Table 1.
  3. [Sec. 4.2, Sec. 4.4] The ablation tables use a subset of 10 PointOdyssey and 20 ScanNet sequences, but it is not stated whether these are the same subsets as in the main evaluation, so the relation between the ablation magnitudes and the final numbers is unclear.
  4. [Sec. 2.2 and Sec. 4.3] The paper claims the method supports videos of 1000 frames or more, but the longest evaluated sequences are 250 frames; a scalability experiment on longer videos would substantiate this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: co-alignment is a prediction-consistency step, all metrics use independent ground truth, and the self-citations to Marigold/MiDaS are reproducible external baselines.

full rationale

The derivation chain is self-contained. The multi-frame LDM is fine-tuned from Marigold, which is a published, code-released prior model with its own external evaluation; citing it is attribution, not load-bearing. The global co-alignment in Eq. 1 and Eq. 3 optimizes consistency among the model's own snippet predictions using only predicted depth values and regularization; it never uses ground-truth depth, so the reported AbsRel/δ1 improvements over baselines are measured on held-out GT under the standard affine-invariant protocol, not forced by construction. The per-snippet inverse-depth normalization in Sec. 3.4 defines training-target normalization, not the evaluation metric. The ablation tables isolate components, and while the per-snippet affine-consistency assumption behind Eq. 3 is an untested robustness risk, an untested assumption is not circularity. No step reduces, by definition or self-citation, to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method has no invented entities. Its free parameters are standard hyperparameters (dilation rates, regularization weights, optimization steps, refinement schedule) that are tuned on a small validation subset and ablations. The central claim rests on domain assumptions: per-snippet scale/shift consistency, the validity of per-snippet inverse-depth normalization, synthetic-to-real transfer, and convergence of the alignment optimizer. These are plausible and partially tested by ablations, but none is independently verified.

free parameters (5)
  • Snippet dilation rates = {1, 10, 25}
    Chosen by hand and ablated in Tab 2; the set of rates determines temporal context and is a load-bearing design choice.
  • Co-alignment regularization weights λ1, λ2 = λ1=1e-1, λ2=1e1
    Set by hand ('work well in practice', Sec A.1); weights control scale/shift prior strength.
  • Co-alignment optimization steps = 2000
    Number of Adam steps used for Eq 3; convergence is assumed, not demonstrated.
  • Refinement schedule = start T/2, 10 steps, dilation 6 to 1
    Manual schedule that balances detail vs. runtime; only optional component.
  • Snippet length n = 3
    Fixed at inference; other lengths are said to be possible but not evaluated.
assumptions (5)
  • domain assumption Each snippet's depth prediction is consistent up to one global scale and shift across its constituent frames.
    Sec 3.3 states scales/shifts are shared across frames of a snippet; if the snippet model introduces intra-snippet inconsistencies, global co-alignment cannot fix them.
  • domain assumption Per-snippet inverse-depth normalization (2nd/98th percentile) preserves relative depth comparability across frames and snippets.
    Sec 3.4 uses this normalization to enable the model to handle rapid depth range changes; its robustness is not independently validated.
  • domain assumption Synthetic training data (TartanAir, Hypersim) transfers zero-shot to real in-the-wild videos.
    The model is fine-tuned only on synthetic data; zero-shot generalization is assumed from Marigold's prior, not measured on held-out real training data.
  • domain assumption The L1 objective in Eq 3 has a usable optimum that prevents drift and is reachable by gradient descent.
    The co-alignment step is the core mechanism; no theoretical guarantee or convergence analysis is given beyond empirical results.
  • domain assumption The Stable Diffusion / Marigold latent diffusion prior remains a valid inverse-depth generator after fine-tuning.
    The method inherits the base model's visual priors and assumes they survive the multi-frame fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video Depth without Video Models." pith.science (2026). https://pith.science/paper/X3M6HZK3

@misc{pith2026241119189,
  author       = {Pith},
  title        = {Pith review of: Video Depth without Video Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3M6HZK3}},
  note         = {Machine review of arXiv:2411.19189}
}
read the original abstract

Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. Recent advances in single-image depth estimation, brought about by the rise of large foundation models and the use of synthetic training data, have fueled a renewed interest in video depth. However, naively applying a single-image depth estimator to every frame of a video disregards temporal continuity, which not only leads to flickering but may also break when camera motion causes sudden changes in depth range. An obvious and principled solution would be to build on top of video foundation models, but these come with their own limitations; including expensive training and inference, imperfect 3D consistency, and stitching routines for the fixed-length (short) outputs. We take a step back and demonstrate how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets (typically frame triplets) to depth snippets. (ii) a robust, optimization-based registration algorithm that optimally assembles depth snippets sampled at various different frame rates back into a consistent video. RollingDepth is able to efficiently handle long videos with hundreds of frames and delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models. Project page: rollingdepth.github.io.

Figures

Figures reproduced from arXiv: 2411.19189 by the authors.

Figure 1
Figure 1. The RollingDepth model takes an unconstrained video and reconstructs a corresponding depth video. Unlike methods that rely on video diffusion models, it extends a single-image monodepth estimator such that it can process short snippets. To account for temporal context, snippets with varying frame rates are sampled from the video, processed, and reassembled through a global alignment algorithm to obtain long, tempora… view at source ↗
Figure 2
Figure 2. Overview of the RollingDepth Inference Pipeline. Given a video sequence x (with i is i th frame), we construct NT overlapping snippets using a dilated rolling kernel with varying dilation rates, and perform 1-step inference to obtain initial depth snippets ( k k k ). Next, depth co-alignment optimizes NT pairs of scale and shift values to achieve globally consistent depth throughout the full video. An optional refin… view at source ↗
Figure 3
Figure 3. Depth Refinement encodes the co-aligned depth video into latent space, contaminates it with a moderate amount of noise, then denoises it with a series of reverse diffusion steps with de￾creasing snippet dilation rate. After each step, overlapping latents are averaged to propagate information between snippets. lation rates, and additional regularization. Once the depth snippets have been aligned in a common frame wit… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison between different methods. RollingDepth excels at preserving fine-grained details (cf . the chandelier in the first sample and the tripod in the third sample) and recovering accurate scene layout (cf . the far plane in the second sample). We filt…
Figure 5
Figure 5. Figure 5: AbsRel error over time: The line plot (left) shows the depth error at every individual frame, end-of-line numbers are the average error across the video. The images (right) display error maps (low high) for two specific frames. RollingDepth achieves the lowest error ov…
Figure 6
Figure 6. Figure 6: Qualitative comparison of depth predictions (near far) from in-the-wild videos. To graphically show temporal con￾sistency, we display temporal profiles (red box) for a fixed column (marked with a red line). RollingDepth picks up subtle details like accessories and wrin…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 47 canonical work pages

  1. [1]

    Bidirectional attention network for monocular depth estimation

    Shubhra Aich, Jean Marie Uwabeza Vianney, Md Amirul Is- lam, Mannat Kaur, and Bingbing Liu. Bidirectional attention network for monocular depth estimation. In ICRA, 2021. 3

    2021

  2. [2]

    AdaBins: Depth estimation using adaptive bins

    Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. AdaBins: Depth estimation using adaptive bins. In CVPR,

  3. [3]

    ZoeDepth: Zero-shot transfer by com- bining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M¨uller. ZoeDepth: Zero-shot transfer by com- bining relative and metric depth. arXiv:2302.12288, 2023. 3

    arXiv 2023

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  5. [5]

    Depth Pro: Sharp monocular metric depth in less than a second

    Aleksei Bochkovskii, Ama ¨el Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth Pro: Sharp monocular metric depth in less than a second. arXiv:2410.02073, 2024. 2, 3

    arXiv 2024

  6. [6]

    Pix2Video: Video editing using image diffusion

    Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2Video: Video editing using image diffusion. In ICCV,

  7. [7]

    Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera

    Yuhua Chen, Cordelia Schmid, and Cristian Sminchis- escu. Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera. In ICCV, 2019. 3

    2019

  8. [8]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Niessner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Niessner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 6

    2017

Show all 88 references
  1. [9]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 12

    2017

  2. [10]

    Warped diffusion: Solving video inverse problems with image diffusion models

    Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola Borislavov Kovachki, and Arash Vahdat. Warped diffusion: Solving video inverse problems with image diffusion models. NeurIPS, 2024. 3

    2024

  3. [11]

    DiffusionDepth: Diffusion denoising approach for monocular depth estima- tion

    Yiqun Duan, Xianda Guo, and Zheng Zhu. DiffusionDepth: Diffusion denoising approach for monocular depth estima- tion. arXiv:2303.05021, 2023. 3

    2023 arXiv

  4. [12]

    Depth map prediction from a single image using a multi-scale deep net- work

    David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep net- work. NeurIPS, 2014. 2

    2014

  5. [13]

    Deep ordinal regression net- work for monocular depth estimation

    Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Bat- manghelich, and Dacheng Tao. Deep ordinal regression net- work for monocular depth estimation. In CVPR, 2018. 3

    2018

  6. [14]

    GeoWiz- ard: Unleashing the diffusion priors for 3d geometry estima- tion from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. GeoWiz- ard: Unleashing the diffusion priors for 3d geometry estima- tion from a single image. In ECCV, 2024. 3

    2024

  7. [15]

    Multi-view stereo: A tutorial

    Yasutaka Furukawa, Carlos Hern ´andez, et al. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015. 2

    2015

  8. [16]

    Fine-tuning image-conditional diffusion models is easier than you think

    Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning image-conditional diffusion models is easier than you think. arXiv:2409.11355, 2024. 3, 12

    2024

  9. [17]

    AliceVision Meshroom: An open- source 3d reconstruction pipeline

    Carsten Griwodz, Simone Gasparini, Lilian Calvet, Pierre Gurdjos, Fabien Castan, Benoit Maujean, Gregoire De Lillo, and Yann Lanthony. AliceVision Meshroom: An open- source 3d reconstruction pipeline. In ACM Multimedia ,

  10. [18]

    DepthFM: Fast monocular depth estimation with flow match- ing

    Ming Gui, Johannes S Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan An- dreas Baumann, Vincent Tao Hu, and Bj ¨orn Ommer. DepthFM: Fast monocular depth estimation with flow match- ing. arXiv:2403.13788, 2024. 3, 4

    2024 arXiv

  11. [19]

    3d packing for self-supervised monocular depth estimation

    Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raven- tos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2020. 13

    2020

  12. [20]

    Towards zero-shot scale-aware monoc- ular depth estimation

    Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares , Ambrus,, and Adrien Gaidon. Towards zero-shot scale-aware monoc- ular depth estimation. In ICCV, 2023. 3

    2023

  13. [21]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Liu, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv:2409.18124, 2024. 3

    2024 arXiv

  14. [22]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. NeurIPS, 2020. 3

    2020

  15. [23]

    Metric3D v2: A versatile monocular geomet- ric foundation model for zero-shot metric depth and surface normal estimation

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3D v2: A versatile monocular geomet- ric foundation model for zero-shot metric depth and surface normal estimation. arXiv:2404.15506, 2024. 3

    2024 arXiv

  16. [24]

    DepthCrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xi- aodong Cun, Yong Zhang, Long Quan, and Ying Shan. DepthCrafter: Generating consistent long depth sequences for open-world videos. arXiv:2409.02095, 2024. 2, 3, 6, 12

    2024 arXiv

  17. [25]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In CVPR, 2024. 2, 3, 4, 6, 12

    2024

  18. [26]

    3d Gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d Gaussian splatting for real-time radiance field rendering. ACM TOG, 42(4):139–1, 2023. 2

    2023

  19. [27]

    Text2Video-Zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2Video-Zero: Text- to-image diffusion models are zero-shot video generators. In ICCV, 2023. 3

    2023

  20. [28]

    EscherNet: A generative model for scalable view synthesis

    Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiao- juan Qi, and Andrew J Davison. EscherNet: A generative model for scalable view synthesis. In CVPR, 2024. 4

    2024

  21. [29]

    Robust consistent video depth estimation

    Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, 2021. 3 9

    2021

  22. [30]

    Solving video inverse problems using image diffusion models

    Taesung Kwon and Jong Chul Ye. Solving video inverse problems using image diffusion models. arXiv:2409.02574,

    arXiv

  23. [31]

    From big to small: Multi-scale local planar guidance for monocular depth estimation

    Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv:1907.10326, 2019. 3

    1907 arXiv

  24. [32]

    MegaDepth: Learning single- view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. MegaDepth: Learning single- view depth prediction from internet photos. In CVPR, 2018. 3

    2018

  25. [33]

    BinsFormer: Revisiting adaptive bins for monocular depth estimation

    Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. BinsFormer: Revisiting adaptive bins for monocular depth estimation. arXiv:2204.00987, 2022. 3

    2022 arXiv

  26. [34]

    DepthFormer: Exploiting long-range correlation and local information for accurate monocular depth estimation

    Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. DepthFormer: Exploiting long-range correlation and local information for accurate monocular depth estimation. Ma- chine Intelligence Research, pages 1–18, 2023. 3

    2023

  27. [35]

    Temporally consistent online depth estimation in dynamic scenes

    Zhaoshuo Li, Wei Ye, Dilin Wang, Francis X Creighton, Russell H Taylor, Ganesh Venkatesh, and Mathias Unberath. Temporally consistent online depth estimation in dynamic scenes. In WACV, 2023. 3

    2023

  28. [36]

    Patch- Fusion: An end-to-end tile-based framework for high- resolution monocular metric depth estimation

    Zhenyu Li, Shariq Farooq Bhat, and Peter Wonka. Patch- Fusion: An end-to-end tile-based framework for high- resolution monocular metric depth estimation. In CVPR,

  29. [37]

    PatchRe- finer: Leveraging synthetic data for real-domain high- resolution monocular metric depth estimation

    Zhenyu Li, Shariq Farooq Bhat, and Peter Wonka. PatchRe- finer: Leveraging synthetic data for real-domain high- resolution monocular metric depth estimation. In ECCV,

  30. [38]

    Common diffusion noise schedules and sample steps are flawed

    Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2024. 12

    2024

  31. [39]

    V A-DepthNet: A variational approach to sin- gle image depth prediction

    Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. V A-DepthNet: A variational approach to sin- gle image depth prediction. In ICLR, 2023. 3

    2023

  32. [40]

    Video-P2P: Video editing with cross-attention control

    Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-P2P: Video editing with cross-attention control. In CVPR, 2024. 3

    2024

  33. [41]

    SyncDreamer: Gen- erating multiview-consistent images from a single-view im- age

    Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDreamer: Gen- erating multiview-consistent images from a single-view im- age. arXiv:2309.03453, 2023. 4

    2023 arXiv

  34. [42]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5

    2019

  35. [43]

    Consistent video depth estimation.ACM Transactions on Graphics, 39(4), 2020

    Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation.ACM Transactions on Graphics, 39(4), 2020. 3

    2020

  36. [44]

    NeRF: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 2

    2021

  37. [45]

    All in tokens: Uni- fying output space of visual tasks via soft token

    Jia Ning, Chen Li, Zheng Zhang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Uni- fying output space of visual tasks via soft token. In ICCV,

  38. [46]

    DINOv2: Learning robust visual features without supervi- sion

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervi- sion. arXiv:2304.07193, 2023. 2, 3

    2023 arXiv

  39. [47]

    ReFusion: 3d reconstruc- tion in dynamic environments for RGB-D cameras exploit- ing residuals

    Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. ReFusion: 3d reconstruc- tion in dynamic environments for RGB-D cameras exploit- ing residuals. In IROS, 2019. 6

    2019

  40. [48]

    P3Depth: Monocular depth estimation with a piecewise planarity prior

    Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3Depth: Monocular depth estimation with a piecewise planarity prior. In CVPR, 2022. 3

    2022

  41. [49]

    UniDepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. In CVPR,

  42. [50]

    UniDepthV2: Universal monocular metric depth estimation made simpler

    Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mat- tia Segu, Siyuan Li, Wim Abbeloos, and Luc Van Gool. UniDepthV2: Universal monocular metric depth estimation made simpler. arXiv preprint arXiv:2502.20110, 2025. 3

    2025 arXiv

  43. [51]

    FateZero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. FateZero: Fus- ing attentions for zero-shot text-based video editing. In ICCV, 2023. 3

    2023

  44. [52]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 2020. 2, 3, 4, 6

    2020

  45. [53]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In ICCV, 2021. 5

    2021

  46. [54]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2, 3

    2022

  47. [55]

    Monocular depth estimation using diffusion models

    Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estimation using diffusion models. arXiv:2302.14816, 2023. 3

    2023 arXiv

  48. [56]

    Structure- from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure- from-motion revisited. In CVPR, 2016. 1

    2016

  49. [57]

    LAION-5B: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. LAION-5B: An open large-scale dataset for train- ing next generation image-text models. NeurIPS, 2022. 3

    2022

  50. [58]

    Learning tem- porally consistent video depth from video diffusion priors

    Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning tem- porally consistent video depth from video diffusion priors. arXiv:2406.01493, 2024. 2, 3, 6, 12

    2024 arXiv

  51. [59]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In ICLR, 2021. 3

    2021

  52. [60]

    Con- sistent direct time-of-flight video depth super-resolution

    Zhanghao Sun, Wei Ye, Jinhui Xiong, Gyeongmin Choe, Jialiang Wang, Shuochen Su, and Rakesh Ranjan. Con- sistent direct time-of-flight video depth super-resolution. arXiv:2211.08658, 2022. 6, 12

    2022 arXiv

  53. [61]

    Deepv2d: Video to depth with differentiable structure from motion

    Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. In ICLR, 2020. 3 10

    2020

  54. [62]

    TartanAir: A dataset to push the limits of visual SLAM

    Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Se- bastian Scherer. TartanAir: A dataset to push the limits of visual SLAM. In IROS, 2020. 5

    2020

  55. [63]

    Less is more: Consistent video depth estimation with masked frames modeling

    Yiran Wang, Zhiyu Pan, Xingyi Li, Zhiguo Cao, Ke Xian, and Jianming Zhang. Less is more: Consistent video depth estimation with masked frames modeling. In ACM MM ,

  56. [64]

    Neural video depth stabilizer

    Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In ICCV, 2023. 3, 6

    2023

  57. [65]

    NVDS+: Towards efficient and versatile neu- ral stabilizer for video depth estimation

    Yiran Wang, Min Shi, Jiaqi Li, Chaoyi Hong, Zihao Huang, Juewen Peng, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. NVDS+: Towards efficient and versatile neu- ral stabilizer for video depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2...

    2024

  58. [66]

    Tune-A-Video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-A-Video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023. 3

    2023

  59. [67]

    Diffusion models trained with large data are transferable vi- sual models

    Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable vi- sual models. arXiv:2403.06090, 2024. 3

    2024 arXiv

  60. [68]

    GMFlow: Learning optical flow via global matching

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. GMFlow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8121– 8130, 2022. 12

    2022

  61. [69]

    Transformer-based attention networks for con- tinuous pixel-wise prediction

    Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-based attention networks for con- tinuous pixel-wise prediction. In ICCV, 2021. 3

    2021

  62. [70]

    Depth any video with scalable synthetic data

    Honghui Yang, Di Huang, Wei Yin, Chunhua Shen, Haifeng Liu, Xiaofei He, Binbin Lin, Wanli Ouyang, and Tong He. Depth any video with scalable synthetic data. arXiv:2410.10815, 2024. 3

    2024 arXiv

  63. [71]

    Depth Anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 3, 6

    2024

  64. [72]

    Depth Any- thing V2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Any- thing V2. arXiv:2406.09414, 2024. 2, 3, 4, 6

    2024 arXiv

  65. [73]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia, 2023. 3

    2023

  66. [74]

    MVSNet: Depth inference for unstructured multi- view stereo

    Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth inference for unstructured multi- view stereo. In ECCV, 2018. 2

    2018

  67. [75]

    MAMo: Leveraging memory and attention for monocular video depth estimation

    Rajeev Yasarla, Hong Cai, Jisoo Jeong, Yunxiao Shi, Risheek Garrepalli, and Fatih Porikli. MAMo: Leveraging memory and attention for monocular video depth estimation. In ICCV, 2023. 3

    2023

  68. [76]

    FutureDepth: Learning to predict the future improves video depth estimation

    Rajeev Yasarla, Manish Kumar Singh, Hong Cai, Yunxiao Shi, Jisoo Jeong, Yinhao Zhu, Shizhong Han, Risheek Gar- repalli, and Fatih Porikli. FutureDepth: Learning to predict the future improves video depth estimation. In European Conference on Computer Vision , pages 440–458. Springer,

  69. [77]

    Di- verseDepth: Affine-invariant depth prediction using diverse data

    Wei Yin, Xinlong Wang, Chunhua Shen, Yifan Liu, Zhi Tian, Songcen Xu, Changming Sun, and Dou Renyin. Di- verseDepth: Affine-invariant depth prediction using diverse data. arXiv:2002.00569, 2020. 3

    2002 arXiv

  70. [78]

    Met- ric3D: Towards zero-shot metric 3d prediction from a single image

    Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Met- ric3D: Towards zero-shot metric 3d prediction from a single image. In ICCV, 2023. 3, 4

    2023

  71. [79]

    NeWCRFs: Neural window fully-connected CRFs for monocular depth estimation

    Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. NeWCRFs: Neural window fully-connected CRFs for monocular depth estimation. In CVPR, 2022. 3

    2022

  72. [80]

    Exploiting temporal consistency for real-time video depth estimation

    Haokui Zhang, Chunhua Shen, Ying Li, Yuanzhouhan Cao, Yu Liu, and Youliang Yan. Exploiting temporal consistency for real-time video depth estimation. In ICCV, 2019. 3

    2019

  73. [81]

    BetterDepth: Plug-and-play dif- fusion refiner for zero-shot monocular depth estimation

    Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Obukhov, Markus Gross, Konrad Schindler, and Christopher Schroers. BetterDepth: Plug-and-play dif- fusion refiner for zero-shot monocular depth estimation. NeurIPS, 2024. 3

    2024

  74. [82]

    Consistent depth of moving objects in video

    Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Consistent depth of moving objects in video. ACM Transactions on Graphics (TOG) , 40(4):1– 12, 2021. 3

    2021

  75. [83]

    Towards consistent video edit- ing with text-to-image diffusion models

    Zicheng Zhang, Bonan Li, Xuecheng Nie, Congying Han, Tiande Guo, and Luoqi Liu. Towards consistent video edit- ing with text-to-image diffusion models. NeurIPS, 2024. 3

    2024

  76. [84]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv:2303.02153, 2023. 3

    2023 arXiv

  77. [85]

    Discrete cosine transform network for guided depth map super-resolution

    Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine transform network for guided depth map super-resolution. In CVPR, pages 5697– 5707, 2022. 3

    2022

  78. [86]

    DDFM: Denoising diffusion model for multi-modality image fusion

    Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. DDFM: Denoising diffusion model for multi-modality image fusion. In ICCV, 2023. 3

    2023

  79. [87]

    PointOdyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J Guibas. PointOdyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV,

  80. [2023]

    num-frames

    5, 12 11 Supplementary Material This supplementary material includes additional implemen- tation details and experimental results. A. Implementation Details A.1. Depth Co-Alignment As discussed in Sec. 3.3, letk(i, j) denote an indexing func- tion that returns the snippet inde...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.