GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth
Pith reviewed 2026-05-20 22:56 UTC · model grok-4.3
The pith
GemDepth embeds predicted camera poses to enforce strict 3D consistency in video depth estimation.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
GemDepth introduces a Geometry-Embedding Module (GEM) that predicts inter-frame camera poses to generate implicit geometric embeddings. These embeddings supply intrinsic 3D perception and alignment. Guided by the embeddings, an Alternating Spatio-Temporal Transformer (ASTT) captures latent point-level correspondences that simultaneously sharpen fine details and enforce rigorous temporal consistency. The framework is trained with a data-efficient strategy and reports state-of-the-art results on multiple datasets, especially in complex dynamic scenes.
What carries the argument
The Geometry-Embedding Module (GEM) that predicts inter-frame camera poses and converts them into implicit geometric embeddings for 3D perception and alignment.
If this is right
- Spatial blurring in fine-detail regions is reduced because point-level correspondences are recovered with geometric guidance.
- Temporal inconsistencies disappear under rotations and drastic view changes once motion priors are injected.
- State-of-the-art depth accuracy holds across multiple public datasets, particularly in dynamic scenes.
- A data-efficient training schedule maintains high performance without requiring massive additional labeled video.
Where Pith is reading between the lines
- The same geometric embeddings could be reused as input features for downstream tasks such as video object segmentation or novel-view synthesis.
- Evaluating the model on camera trajectories with extreme angular velocity not present in current benchmarks would test how far the 3D consistency extends.
- Real-time robotics or AR pipelines that already estimate camera pose could adopt the embeddings with little extra cost.
Load-bearing premise
Video depth networks achieve strict 3D geometric consistency only when they receive explicit awareness of camera motion and global 3D structure.
What would settle it
On a test set of video sequences containing known large rotations or sudden viewpoint changes, remove the Geometry-Embedding Module and measure whether 3D consistency metrics drop significantly compared with the full model.
Figures
read the original abstract
Video depth estimation extends monocular prediction into the temporal domain to ensure coherence. However, existing methods often suffer from spatial blurring in fine-detail regions and temporal inconsistencies. We argue that current approaches, which primarily rely on temporal smoothing via Transformers, struggle to maintain strict 3D geometric consistency-particularly under rotations or drastic view changes. To address this, we propose GemDepth, a framework built on the insight that an explicit awareness of camera motion and global 3D structure is a prerequisite for 3D consistency. Distinctively, GemDepth introduces a Geometry-Embedding Module (GEM) that predicts inter-frame camera poses to generate implicit geometric embeddings. This injection of motion priors equips the network with intrinsic 3D perception and alignment capabilities. Guided by these geometric cues, our Alternating Spatio-Temporal Transformer (ASTT) captures latent point-level correspondences to simultaneously enhance spatial precision for sharp details and enforce rigorous temporal consistency. Furthermore, GemDepth employs a data-efficient training strategy, effectively bridging the gap between high efficiency and robust geometric consistency. As shown in Fig.2, comprehensive evaluations demonstrate that GemDepth achieves state-of-the-art performance across multiple datasets, particularly in complex dynamic scenarios. The code is publicly available at: https://github.com/Yuecheng919/GemDepth.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GemDepth, a video depth estimation framework that introduces a Geometry-Embedding Module (GEM) to predict inter-frame camera poses and generate implicit geometric embeddings. These embeddings guide an Alternating Spatio-Temporal Transformer (ASTT) to capture latent point-level correspondences, aiming to improve spatial precision for fine details and enforce rigorous temporal consistency. The method employs a data-efficient training strategy and claims state-of-the-art performance across multiple datasets, particularly in complex dynamic scenarios.
Significance. If the results hold, the explicit incorporation of camera motion priors for intrinsic 3D perception represents a meaningful architectural shift from purely transformer-based temporal smoothing. This could improve robustness under rotations and drastic view changes. Public code availability aids reproducibility and allows direct verification of the geometric consistency claims.
major comments (2)
- [GEM description and experiments] The central claim of strict 3D geometric consistency in complex dynamic scenarios rests on the reliability of pose predictions from the Geometry-Embedding Module (GEM). However, moving objects violate the static-scene assumption underlying monocular pose estimation, which could render the injected motion priors ineffective. The manuscript should include quantitative pose error metrics (rotation/translation) and ablations isolating GEM's contribution on dynamic subsets of the evaluated datasets.
- [Evaluation and results] The abstract asserts SOTA results and highlights performance in dynamic scenarios, yet supplies no quantitative metrics, error bars, dataset details, or ablation tables. Without these in the full manuscript (e.g., in the evaluation section or Table 1/2), it is impossible to assess whether the data support the geometric-consistency argument over standard depth metrics.
minor comments (2)
- [Figure 2] Figure 2 is referenced for comprehensive evaluations but lacks a clear caption or legend explaining the visualized consistency metrics.
- [Method overview] Notation for the implicit geometric embeddings and how they are injected into ASTT could be formalized with an equation or diagram for clarity.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments. We address each major comment below and describe the revisions planned for the next version of the manuscript.
read point-by-point responses
-
Referee: [GEM description and experiments] The central claim of strict 3D geometric consistency in complex dynamic scenarios rests on the reliability of pose predictions from the Geometry-Embedding Module (GEM). However, moving objects violate the static-scene assumption underlying monocular pose estimation, which could render the injected motion priors ineffective. The manuscript should include quantitative pose error metrics (rotation/translation) and ablations isolating GEM's contribution on dynamic subsets of the evaluated datasets.
Authors: We agree that explicit validation of the pose predictions is important, particularly under dynamic conditions where the static-scene assumption may be violated. Although GEM is trained end-to-end to produce geometric embeddings that support 3D consistency, we will add quantitative pose error metrics (rotation and translation errors) on the evaluated datasets. We will also include new ablations that isolate GEM's contribution specifically on dynamic subsets to directly address this concern. revision: yes
-
Referee: [Evaluation and results] The abstract asserts SOTA results and highlights performance in dynamic scenarios, yet supplies no quantitative metrics, error bars, dataset details, or ablation tables. Without these in the full manuscript (e.g., in the evaluation section or Table 1/2), it is impossible to assess whether the data support the geometric-consistency argument over standard depth metrics.
Authors: The full manuscript already reports quantitative results, including standard depth metrics, error bars, dataset specifications, and ablation studies in the evaluation section and associated tables. To strengthen the link between these results and the geometric-consistency claims, we will expand the discussion of dynamic scenarios, add explicit references to the relevant tables in the abstract and introduction, and include additional breakdowns on dynamic subsets. revision: partial
Circularity Check
No circularity: architectural design and evaluation remain independent of inputs
full rationale
The paper advances GemDepth via an explicit design insight that camera-motion awareness (via GEM pose prediction) is a prerequisite for 3D consistency, implemented in GEM and ASTT modules, then validated on external datasets. No equation, parameter fit, or self-citation reduces the claimed consistency to a quantity chosen on the same data or to a prior result by the same authors. The derivation chain from insight to modules to reported metrics is self-contained and does not collapse by construction.
Axiom & Free-Parameter Ledger
Lean theorems connected to this paper
-
IndisputableMonolith/Foundation/AlexanderDuality.leanalexander_duality_circle_linking unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
Geometry-Embedding Module (GEM) that predicts inter-frame camera poses to generate implicit geometric embeddings... explicit awareness of camera motion and global 3D structure is a prerequisite for 3D consistency
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Frozen in time: A joint video and image encoder for end-to-end retrieval
Birkl, R., Wofk, D., and M¨uller, M. Midas v3. 1–a model zoo for robust monocular relative depth estimation.arXiv preprint arXiv:2307.14460,
-
[2]
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y ., English, Z., V oleti, V ., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023a. Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align you...
work page internal anchor Pith review Pith/arXiv arXiv
-
[3]
arXiv preprint arXiv:2001.10773,
work page internal anchor Pith review Pith/arXiv arXiv 2001
-
[4]
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
Chen, H., Xia, M., He, Y ., Zhang, Y ., Cun, X., Yang, S., Xing, J., Liu, Y ., Chen, Q., Wang, X., et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512,
work page internal anchor Pith review Pith/arXiv arXiv
-
[5]
Romeo: Robust metric visual odometry
Cheng, J., Cai, Z., Zhang, Z., Yin, W., Muller, M., Paulitsch, M., and Yang, X. Romeo: Robust metric visual odometry. arXiv preprint arXiv:2412.11530, 2024a. Cheng, J., Xu, G., Guo, P., and Yang, X. Coatrsnet: Fully exploiting convolution and attention for stereo matching by region separation.International Journal of Computer Vision, 132(1):56–73, 2024b. ...
-
[6]
Deep ordinal regression network for monocular depth estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. Deep ordinal regression network for monocular depth estimation. InProceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002–2011,
work page 2002
-
[7]
Depthcrafter: Generating consistent long depth sequences for open-world videos
Hu, W., Gao, X., Li, X., Zhao, S., Cun, X., Zhang, Y ., Quan, L., and Shan, Y . Depthcrafter: Generating consistent long depth sequences for open-world videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 2005–2015,
work page 2005
-
[8]
MapAnything: Universal Feed-Forward Metric 3D Reconstruction
Keetha, N., M¨uller, N., Sch¨onberger, J., Porzi, L., Zhang, Y ., Fischer, T., Knapitsch, A., Zauss, D., Weber, E., Antunes, N., et al. Mapanything: Universal feed-forward metric 3d reconstruction.arXiv preprint arXiv:2509.13414,
work page internal anchor Pith review Pith/arXiv arXiv
-
[9]
Depth Anything 3: Recovering the Visual Space from Any Views
Lin, H., Chen, S., Liew, J. H., Chen, D. Y ., Li, Z., Shi, G., Feng, J., and Kang, B. Depth anything 3: Recov- ering the visual space from any views.arXiv preprint arXiv:2511.10647,
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
Stabledpt: Temporal stable monocular video depth estimation.arXiv preprint arXiv:2601.02793,
Sobko, I., Riemenschneider, H., Gross, M., and Schroers, C. Stabledpt: Temporal stable monocular video depth estimation.arXiv preprint arXiv:2601.02793,
-
[12]
Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., and Novotny, D. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294–5306, 2025a. Wang, Q., Zheng, S., Yan, Q., Deng, F., Zhao, K., and Chu, X. Irs: A large naturalistic indoor robotics stereo dataset to train deep models fo...
-
[13]
Stereogen: High-quality stereo image generation from a single image.arXiv e- prints, pp
Wang, X., Yang, H., Xu, G., Cheng, J., Lin, M., Deng, Y ., Zang, J., Chen, Y ., and Yang, X. Stereogen: High-quality stereo image generation from a single image.arXiv e- prints, pp. arXiv–2501, 2025b. Wang, X., Yang, H., Xu, G., Cheng, J., Lin, M., Deng, Y ., Zang, J., Chen, Y ., and Yang, X. Zerostereo: Zero-shot stereo matching from single images. InPro...
-
[14]
Xu, G., Lin, H., Luo, H., Wang, X., Yao, J., Zhu, L., Pu, Y ., Chi, C., Sun, H., Wang, B., et al. Pixel-perfect depth with semantics-prompted diffusion transformers.arXiv preprint arXiv:2510.07316, 2025a. Xu, G., Liu, J., Wang, X., Cheng, J., Deng, Y ., Zang, J., Chen, Y ., and Yang, X. Banet: Bilateral aggregation network for mobile stereo matching. InPr...
-
[15]
Table 8.Summary of datasets used for training
This approach enables the network to generalize across varying aspect ratios and resolutions. Table 8.Summary of datasets used for training. Dataset Indoor Outdoor # Images Pose-annotated datasets Virtual KITTI 2 (Cabon et al., 2020)✓40K Tartanair (Wang et al., 2020)✓ ✓300K PointOdyssey (Zheng et al., 2023)✓ ✓70K MVS-Synth (Huang et al., 2018)✓80K Dynamic...
work page 2020
-
[16]
dataset as a benchmark. Specifically, we conduct a comparative analysis between two versions of our GemDepth and the current state-of-the-art method, VideoDepthAnything (VDA) (Chen et al., 2025), under five different sequence lengths: 500, 400, 300, 200, and 100 frames. As demonstrated in Tab. 10, both versions of GemDepth consistently outperform VDA acro...
work page 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.