Pith. sign in

REVIEW 1 major objections 1 minor 1 cited by

Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors

T0 review · 1 major / 1 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video depth and normal maps can be learned from unlabeled video by keeping a single-image model frozen and adding an optical-flow smoothness loss.

desk verdict A genuinely useful zero-shot recipe for video depth/normal that beats its image backbones and approaches video-supervised models, but the temporal-consistency metric mirrors the training loss and the warp loss enforces an oversimplified geometry. read the letter →

arxiv 2411.17249 v1 pith:CX3JWD2J submitted 2024-11-26 cs.CV cs.AI

Zhengfei Kuang , Tianyuan Zhang , Kai Zhang , Hao Tan , Sai Bi , Yiwei Hu , Zexiang Xu , Milos Hasan
show 2 more authors
Gordon Wetzstein Fujun Luan
This is my paper · ORCID
classification cs.CVcs.AI
keywords videodepthestimationsurfacenormalgeometricbufferszero-shottrainingopticalflowtemporalconsistencysingle-imagepriorsattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that temporally consistent video depth and surface-normal maps, which the authors call geometric buffers, can be trained without any paired video-geometry supervision. The recipe freezes a strong single-image estimator, inserts lightweight temporal attention blocks, and fine-tunes only those blocks under a hybrid loss: a regularization term keeps each frame close to the image model's prediction, and an optical-flow-based term asks that pixels matched between adjacent frames receive the same predicted value. Applied to Depth Anything V2 for depth and Marigold-E2E-FT for normals, the method improves temporal consistency over the image backbones while keeping per-frame accuracy, and its depth results are comparable to a state-of-the-art video model trained on large-scale paired video data. The practical stake is that expensive video-depth and video-normal annotation may be replaceable by unlabeled video plus optical flow.

What carries the argument

The load-bearing object is the hybrid loss, L = ω_reg L_reg + L_stable, computed while the video model processes K frames. L_reg is an affine-invariant relative depth loss (or an L2 loss on latent normal maps) against the fixed image model, which anchors each frame to the image prior; L_stable warps the predicted buffer from one frame to the next using a pretrained optical-flow model and penalizes L1 differences at flow-corresponding pixels, in both forward and backward directions. Because optical flow is imperfect, only pixels that pass cycle validation and are not near Canny edges of the predicted buffer contribute to L_stable. The temporal architecture injects zero-initialized temporal attention blocks between frozen layers of the backbone, so training starts from the image model and adds cross-frame reasoning; for the normal model, deferred back-propagation keeps the stabilization loss computable by decoding chunks of latent maps.

What would settle it

A decisive test is to render a synthetic video of a static scene with known ground-truth depth and normals while the camera translates sideways and rotates, then check whether the model's predictions at flow-corresponding pixels are biased toward equal values by an amount that grows with inter-frame motion and object depth.

Watch

Extended reading notes

Core claim

The central claim is that the temporal inconsistency of image-based geometry models is not a defect that requires video ground truth to fix. A video model can be produced by taking an image prior, freezing most of its weights, inserting temporal attention layers that start from zero, and training with a loss that both preserves the image model's per-frame predictions and pulls flow-corresponding pixels toward equal depth or normal values. On the DepthCrafter benchmark, this zero-shot model improves on Depth Anything V2 on AbsRel, δ1, and OPW across ScanNet, KITTI, and Bonn, and reaches accuracy and smoothness comparable to video-supervised models such as DepthCrafter despite seeing no paired video-depth data. For normals, per-frame angular errors stay close to Marigold-E2E-FT while temporal smoothness improves sharply, e.g., OPW drops from 0.152 to 0.065 on Sintel and 0.092 to 0.069 on ScanNet.

Load-bearing premise

The load-bearing premise is that two pixels linked by optical flow between adjacent frames should be assigned the same depth value or the same normal vector, which is exactly true only for a static scene seen without rotation; under camera translation the true depth at corresponding pixels differs, and under rotation the true normal direction changes, so the stabilization target is an approximation rather than a geometric ground truth.

Editorial extensions

If this is right

  • Temporal consistency of video depth improves without sacrificing accuracy: on KITTI, OPW drops from 0.089 for Depth Anything V2 to 0.038 for the fine-tuned video model, with AbsRel also improving.
  • Zero-shot training reaches parity with paired-data video methods: on ScanNet and KITTI, the depth model is comparable to DepthCrafter in accuracy and better in OPW, despite never seeing video-depth ground truth.
  • Video normal estimation becomes a self-supervised problem: per-frame angular accuracy stays near the Marigold-E2E-FT backbone while temporal smoothness improves on both Sintel and ScanNet.
  • Training geometry-video models is cheap enough to scale: roughly one day on 24 H100 GPUs, because the backbone is frozen and only temporal layers plus a light fine-tuning head are trained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same skeleton should transfer to other buffer types the paper lists but does not test, such as lighting or material maps, as long as a strong single-image estimator exists and the regularization term is adapted to that buffer.
  • Beyond the paper: a 3D-aware variant of the stabilization loss, one that warps using estimated camera motion or rigid scene flow instead of raw optical flow, would isolate how much of the remaining temporal error comes from the warp-invariance approximation.
  • Beyond the paper: because only RGB video and optical flow are needed, collecting more unlabeled video should translate into better temporal consistency at no annotation cost, opening a scaling route that paired-data video models do not have.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper proposes Buffer Anytime, a training framework that converts a single-image depth or normal estimator into a video estimator without paired video-geometry ground truth. The method fine-tunes an image backbone (Depth Anything V2 for depth, Marigold-E2E-FT for normals) with lightweight temporal attention layers, using a hybrid loss: a per-frame regularization loss that keeps predictions aligned with the frozen image model, and an optical-flow-based stabilization loss that penalizes differences between neighboring frames at flow-corresponded pixels. Training is performed on roughly 200K unlabeled videos. Experiments on ScanNet, KITTI, Bonn, and Sintel report improved temporal consistency (OPW) and maintained or slightly improved per-frame accuracy relative to the image baselines, with depth results claimed to be comparable to video-trained methods such as DepthCrafter.

Significance. If the main claim is validated, the framework is practically significant: it offers a recipe for upgrading strong single-image geometry models to temporally coherent video predictors while bypassing expensive paired video-depth and video-normal datasets. The paper is careful in several respects: it evaluates on multiple standard benchmarks, reports ablations of regularization weight and masking, provides pseudocode for the deferred back-propagation and stabilization losses, and explicitly acknowledges limitations of the backbone and of adjacent-frame-only flow guidance. The core idea of combining image priors with optical-flow stabilization is appealing and likely to be influential. The central unaddressed risk is that the temporal-consistency objective is not geometrically correct under camera motion; because the evaluation metric OPW is defined similarly to the training loss, the reported consistency gains may partly reflect optimization of a non-geometric target.

major comments (1)
  1. [Sec. 4.3 and Supp. Table 4] The ablation studies do not isolate the contribution of the temporal attention module from the contribution of the stabilization loss. In Supp. Table 4, "Ours w/o fine-tuning" (temporal layers trained with the frozen backbone refinement network) already gives 0.121 AbsRel and 0.040 OPW, close to the full model's 0.119 and 0.038, and "Ours all frames" gives 0.120/0.865/0.035, slightly better OPW than the full model's 0.038. The manuscript would be strengthened by a variant that uses the same stabilization loss on the image backbone without any temporal attention, and a variant with temporal attention but without L_stable, so the reader can see which component drives the reported gains.
minor comments (1)
  1. [Supp. Sec. 7] The main text says the training process takes approximately one day for 20,000 iterations, while the supplementary says approximately one day for 15,000 iterations. These numbers should be reconciled.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the temporal-consistency metric (OPW) is defined similarly to the optical-flow stabilization loss, so OPW gains reflect the training objective; per-frame accuracy metrics remain externally grounded.

  1. self definitional [Algorithm 2 (Sec. 7.2) and Sec. 4.1 (metrics)]
    "Lf wd stable ← 1 (K−1)HW · |(Warp(Gpred 1,...,K−1, Of wd) − Gpred 2,...,K ) · Mf wd|1; ... Lstable ← 1 2 (Lf wd stable + Lbwd stable); ... We report three metrics: Mean Absolute Relative Error ( AbsRel), the percentage of pixels within 1.25× of ground truth (δ1), and optical-flow-based smoothness error (OPW), defined similarly to our smoothness loss."

    The training objective L_stable is an optical-flow warp-invariance loss between adjacent predicted geometric buffers. The paper then reports OPW, an optical-flow-based smoothness error, as its temporal-consistency metric and explicitly states that OPW is 'defined similarly to our smoothness loss.' Thus the OPW reductions in Tables 1-2 are expected by construction from minimizing L_stable; they do not independently establish that the predicted depth or normal maps are geometrically consistent across camera motion. The temporal-consistency portion of the central claim therefore reduces to the training objective, while the per-frame accuracy claims (AbsRel, δ1, angular errors) remain supported by external ground-truth benchmarks.

full rationale

The paper's core geometric outputs are not derived from the evaluation labels, and per-frame accuracy is measured against external datasets (ScanNet, KITTI, Bonn, Sintel), so the method is not globally circular. The deferred back-propagation self-citation (Zhang et al. [60]) is an implementation detail, not a load-bearing premise. No uniqueness theorem or ansatz is imported from the authors' prior work to force the central design. The substantive circularity is confined to the temporal-consistency evaluation: the training loss L_stable is an optical-flow warp error on consecutive predicted buffers, and the reported OPW metric is described as 'defined similarly to our smoothness loss.' Consequently, the headline temporal-consistency improvements are partly a measure of how well the model minimized its own objective, not a fully independent verification. The limitation discussion in Sec. 5 (e.g., flow only captures adjacent-frame correlations) further acknowledges that the stabilization signal is local. Overall, this is partial circularity in one of the two major evaluation claims, with independent content remaining in the accuracy metrics and qualitative comparisons.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central result rests on several unstated domain assumptions: optical-flow warp-invariance of depth/normal values, reliability of the frozen image prior, reliability of the optical flow model, and representativeness of the unspecified 200K-video training set. The hyperparameters ω_reg, τ_c, edge-mask dilation, and chunk size are hand-chosen and only partially ablated. No new physical entities are introduced; temporal attention blocks are architectural components, not postulated entities.

free parameters (4)
  • regularization weight ω_reg = 1 (ablated 0.1 and 3)
    Balances per-frame image-prior alignment against optical-flow stabilization in Eq 3; set to 1 across experiments.
  • cycle-validation threshold τ_c = 0.34 (log(2)/2)
    Filters optical-flow outliers in Eq 6; not ablated in the paper.
  • edge-mask dilation size = 3 pixels (Manhattan distance)
    Defines the Canny edge exclusion band around predicted depth edges in Sec 3.2; hand-set, not ablated.
  • deferred back-propagation chunk size C = 4 frames
    Chunk size for normal-model stabilization loss in Sec 3.1 to manage memory; affects gradient approximation.
assumptions (5)
  • domain assumption A pixel and its optical-flow-correspondent pixel in an adjacent frame should have equal predicted depth/normal values after warping.
    Invoked by Eqs 4-5 and Algorithm 2 for L_stable. Not geometrically valid under camera translation (depth changes) or rotation (normal direction changes); treated as an approximation with edge/cycle masking.
  • domain assumption The pretrained optical flow model returns accurate correspondences on general unlabeled videos.
    The training loss depends entirely on optical flow quality (Sec 3.2); only cycle validation and edge masking mitigate failures.
  • domain assumption The frozen single-image model's predictions are a trustworthy per-frame target for the video model.
    Regularization losses Eqs 1-2 anchor the video model to Depth Anything V2 / Marigold-E2E-FT; if these image priors fail in an extreme case, the video model inherits the failure (acknowledged in Sec 5).
  • domain assumption The approximately 200K unlabeled training videos are representative of ScanNet, KITTI, Bonn, and Sintel test distributions.
    Sec 4 and Supp Sec 7 give no source or curation details for the training corpus, so distribution shift is unquantified.
  • domain assumption Canny edge detection on predicted depth/normal maps and cycle validation remove the pixels where the warp loss would be invalid.
    Sec 3.2 relies on these masks to curate 'correctly corresponded pixels'; no analysis is provided for occluded or textureless regions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors." pith.science (2026). https://pith.science/paper/CX3JWD2J

@misc{pith2026241117249,
  author       = {Pith},
  title        = {Pith review of: Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CX3JWD2J}},
  note         = {Machine review of arXiv:2411.17249}
}
read the original abstract

We present Buffer Anytime, a framework for estimation of depth and normal maps (which we call geometric buffers) from video that eliminates the need for paired video--depth and video--normal training data. Instead of relying on large-scale annotated video datasets, we demonstrate high-quality video buffer estimation by leveraging single-image priors with temporal consistency constraints. Our zero-shot training strategy combines state-of-the-art image estimation models based on optical flow smoothness through a hybrid loss function, implemented via a lightweight temporal attention architecture. Applied to leading image models like Depth Anything V2 and Marigold-E2E-FT, our approach significantly improves temporal consistency while maintaining accuracy. Experiments show that our method not only outperforms image-based approaches but also achieves results comparable to state-of-the-art video models trained on large-scale paired video datasets, despite using no such paired video data.

Figures

Figures reproduced from arXiv: 2411.17249 by the authors.

Figure 1
Figure 1. Buffer Anytime improves temporal consistency in video geometry estimation without paired training data. Top: Comparison of depth estimation between Depth Anything V2 [56] and our method on a challenging dynamic scene with lighting variations. While the original model shows inconsistent depth predictions across frames, our approach maintains stable depth estimates. Bottom: Surface normal estimation comparison between… view at source ↗
Figure 2
Figure 2. Visualization of Our Training Pipeline. Our pipeline consists of three branches: an optical flow network that extracts optical flow from input video to guide temporal smoothness; a fixed single-frame image model for regularization, and the trained video model that integrates a fine-tuned image backbone with tem￾poral layers. The prediction of f video θ should satisfy two conditions: First, each frame of the depth pr… view at source ↗
Figure 3
Figure 3. Illustration of our masking procedure for the optical flow loss. Row 1: Given two adjacent frames, we first apply cycle validation on the predicted optical flows to filter out the outliers; Row 2: We then apply an edge detection procedure on the pre￾dicted depth map to mask out the boundaries. Row 3: The combi￾nation of two masks diminish the effect of inaccurate optical flow prediction to the smoothness error map. … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Our Network Architecture. We present two model architectures for video geometry estimation: (a) A depth estimation model based on Depth Anything V2 [56], where we inject temporal blocks between fusion layers while keeping the ViT backbone frozen. The model processes vi…
Figure 5
Figure 5. Figure 5: Qualitative comparison on Video Depth Estimation. For better visualization, we also show the time slice on the red lines of each video on their right side. Our model keeps the structure details shown in the image model results while achieving smoother performance on th…
Figure 6
Figure 6. Figure 6: Qualitative comparison on Video Normal Estimation. We show the same time slice as in the depth estimation results, and two predicted frames of each model corresponding to the input frames on the first and third lines. Our model successfully removes the inconsistency fr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video Generation Models are General-Purpose Vision Learners

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A video-diffusion backbone fine-tuned as a single-step multi-task perceiver matches or beats specialists on depth, normals, pose and segmentation, with high data efficiency and sim-to-real transfer.

Reference graph

Works this paper leans on

64 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Stable diffusion version 2

    Stability AI. Stable diffusion version 2. https : / / huggingface . co / stabilityai / stable - diffusion-2, 2022. Accessed: 2024-11-11. 2, 3, 5

    work page 2022

  2. [2]

    Rethinking induc- tive biases for surface normal estimation

    Gwangbin Bae and Andrew J Davison. Rethinking induc- tive biases for surface normal estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9535–9545, 2024. 3, 6

    work page 2024

  3. [3]

    Es- timating and exploiting the aleatoric uncertainty in surface normal estimation

    Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Es- timating and exploiting the aleatoric uncertainty in surface normal estimation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 13137–13146,

  4. [4]

    Marr revisited: 2d-3d alignment via surface normal prediction

    Aayush Bansal, Bryan Russell, and Abhinav Gupta. Marr revisited: 2d-3d alignment via surface normal prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5965–5974, 2016. 3

    work page 2016

  5. [5]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2, 3

    arXiv 2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  7. [7]

    A naturalistic open source movie for opti- cal flow evaluation

    Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for opti- cal flow evaluation. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12 , pages 611–

    2012

  8. [8]

    A computational approach to edge detection

    John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelli- gence, (6):679–698, 1986. 4

    1986

Show all 64 references
  1. [9]

    Learning structure affinity for video depth estima- tion

    Yuanzhouhan Cao, Yidong Li, Haokui Zhang, Chao Ren, and Yifan Liu. Learning structure affinity for video depth estima- tion. In Proceedings of the 29th ACM International Confer- ence on Multimedia , page 190–198, New York, NY , USA,

  2. [10]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion,

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mo- hammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion,

  3. [11]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 6

    2017

  4. [12]

    Surface normal estimation of tilted images via spatial rectifier

    Tien Do, Khiem Vuong, Stergios I Roumeliotis, and Hyun Soo Park. Surface normal estimation of tilted images via spatial rectifier. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 265–280. Springer, 2020. 3

    2020

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Omnidata: A scalable pipeline for making multi- task mid-level vision datasets from 3d scans

    Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi- task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 10786–10796, 2021. 3

    2021

  7. [15]

    Predicting depth, surface nor- mals and semantic labels with a common multi-scale con- volutional architecture

    David Eigen and Rob Fergus. Predicting depth, surface nor- mals and semantic labels with a common multi-scale con- volutional architecture. In Proceedings of the IEEE inter- national conference on computer vision , pages 2650–2658,

  8. [16]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30: 681–694, 2020. 2

    2020

  9. [17]

    Unfolding an indoor origami world

    David Ford Fouhey, Abhinav Gupta, and Martial Hebert. Unfolding an indoor origami world. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VI 13, pages 687–702. Springer, 2014. 3

    2014

  10. [18]

    Deep ordinal regression net- work for monocular depth estimation

    Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Bat- manghelich, and Dacheng Tao. Deep ordinal regression net- work for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2002–2011, 2018. 2

    2002

  11. [19]

    Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image. In European Conference on Computer Vision, pages 241–258. Springer, 2025. 2, 3

    2025

  12. [20]

    Fine-tuning image-conditional diffusion models is easier than you think

    Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning image-conditional diffusion models is easier than you think. arXiv preprint arXiv:2409.11355 ,

  13. [21]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The Inter- national Journal of Robotics Research , 32(11):1231–1237,

  14. [22]

    Unsupervised monocular depth estimation with left- right consistency

    Cl ´ement Godard, Oisin Mac Aodha, and Gabriel J Bros- tow. Unsupervised monocular depth estimation with left- right consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270–279,

  15. [23]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. In arXiv, 2023. 3

    2023

  16. [24]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 3, 5

    2023 arXiv

  17. [25]

    Cameractrl: Enabling camera control for text-to-video generation, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation, 2024. 3

    2024

  18. [26]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Liu, Bingbing Liu, and Ying- Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409.18124, 2024. 2, 3, 6

    2024 arXiv

  19. [27]

    Au- tomatic photo pop-up

    Derek Hoiem, Alexei A Efros, and Martial Hebert. Au- tomatic photo pop-up. In ACM SIGGRAPH 2005 Papers , pages 577–584. 2005. 3

    2005

  20. [28]

    Recov- ering surface layout from an image

    Derek Hoiem, Alexei A Efros, and Martial Hebert. Recov- ering surface layout from an image. International Journal of Computer Vision, 75:151–172, 2007. 2, 3

    2007

  21. [29]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In arXiv,

  22. [30]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 2, 6, 1

    2024 arXiv

  23. [31]

    3d common corruptions and data augmentation

    O ˘guzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18963–18974, 2022. 3

    2022

  24. [32]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9492...

    2024

  25. [33]

    Wetzstein

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hong- sheng Li, Leonidas Guibas, and Gordon. Wetzstein. Collab- orative video diffusion: Consistent multi-video generation with camera control. In arXiv, 2024. 3

    2024

  26. [34]

    Sift flow: Dense correspondence across different scenes

    Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T Freeman. Sift flow: Dense correspondence across different scenes. In Computer Vision–ECCV 2008: 10th Eu- ropean Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10, pag...

    2008

  27. [35]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 5, 1

    2019

  28. [36]

    Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting resid- uals

    Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting resid- uals. In 2019 IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pages 78...

    2019

  29. [37]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. 1

    2017

  30. [38]

    State of the art on diffusion models for visual computing

    Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art on diffusion models for visual computing. In Computer Graphics Forum, page e15063. Wiley Online Library, 2024. 2

    2024

  31. [39]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  32. [40]

    Geonet: Geometric neural network for joint depth and surface normal estimation

    Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 283–291, 2018. 3

    2018

  33. [41]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020. 2

    2020

  34. [42]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021. 4

    2021

  35. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 5

    2022

  36. [44]

    Make3d: Learning 3d scene structure from a single still image

    Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence , 31(5):824–840, 2008. 2

    2008

  37. [45]

    Learning tem- porally consistent video depth from video diffusion priors

    Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning tem- porally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.01493, 2024. 2, 6

    2024 arXiv

  38. [46]

    Human4dit: 360-degree human video gen- eration with 4d diffusion transformer

    Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: 360-degree human video gen- eration with 4d diffusion transformer. ACM Transactions on Graphics (TOG), 43(6), 2024. 3

    2024

  39. [47]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  40. [48]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  41. [49]

    Diffusers: State-of-the-art diffu- sion models

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffu- sion models. https://github.com/huggingface/ diffusers...

    2022

  42. [50]

    Vplnet: Deep single view normal estimation with vanishing points and lines

    Rui Wang, David Geraghty, Kevin Matzen, Richard Szeliski, and Jan-Michael Frahm. Vplnet: Deep single view normal estimation with vanishing points and lines. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 689–698, 2020. 3

    2020

  43. [51]

    De- signing deep networks for surface normal estimation

    Xiaolong Wang, David Fouhey, and Abhinav Gupta. De- signing deep networks for surface normal estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 539–547, 2015. 3

    2015

  44. [52]

    Less is more: Consistent video depth estimation with masked frames modeling

    Yiran Wang, Zhiyu Pan, Xingyi Li, Zhiguo Cao, Ke Xian, and Jianming Zhang. Less is more: Consistent video depth estimation with masked frames modeling. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6347–6358, 2022. 4

    2022

  45. [53]

    Neural video depth stabilizer

    Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 9466–9476,

  46. [54]

    Camco: Camera- controllable 3d-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera- controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024. 3

    2024 arXiv

  47. [55]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2, 3

    2024

  48. [56]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 1, 2, 4, 5, 6

    2024 arXiv

  49. [57]

    En- forcing geometric constraints of virtual normal for depth pre- diction

    Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. En- forcing geometric constraints of virtual normal for depth pre- diction. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 5684–5693, 2019. 2

    2019

  50. [58]

    Rgb ↔x: Image decomposition and synthesis using material-and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloˇs Haˇsan. Rgb ↔x: Image decomposition and synthesis using material-and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 2

    2024

  51. [59]

    Hierarchical normalization for robust monocular depth estimation

    Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. Advances in Neural Informa- tion Processing Systems, 35:14128–14139, 2022. 2

    2022

  52. [60]

    Arf: Artistic radiance fields

    Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In European Conference on Computer Vision, pages 717–733. Springer, 2022. 4, 1 Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors Supplementa...

    2022

  53. [62]

    More Video Results In addition to the qualitative comparisons in the paper, we provide more animated results in our supplementary web- site for better visualization of the prediction quality

  54. [63]

    We utilize the official implementations of Depth Anything V2 [56] and Marigold-E2E-FT [20], adapting temporal blocks from the UnetMotion architecture in the Diffusers [49] library

    More Implementation Details All models are implemented in PyTorch [37]. We utilize the official implementations of Depth Anything V2 [56] and Marigold-E2E-FT [20], adapting temporal blocks from the UnetMotion architecture in the Diffusers [49] library. Experiments are conducte...

  55. [64]

    Additional Ablation Studies We extend our ablation studies beyond the main paper by comparing our model with additional variants: Model with L1 replaces L2 with L1 for the affine-invariant rela- tive loss in the depth model; Model w/o fine-tuning main- tains a fixed refinement...

  56. [2021]

    Association for Computing Machinery. 4

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.