REVIEW 4 major objections 7 minor 82 references
Geometry-Grounded Unified 3D Perception for Autonomous Driving
T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read One shared geometry latent powers state-of-the-art 3D detection, occupancy, and depth for camera-only driving perception.
desk verdict A serious and well-executed attempt to make a reconstruction-pretrained latent serve as a shared 3D perception backbone, but the headline SOTA claims are underdetermined by confounded multi-dataset training and an unclosed ablation gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a 12-block subset of VGGT's reconstruction-pretrained transformer backbone, called VGGT-12. It receives image patch tokens from a DINOv2 encoder, augmented with Plücker raymap embeddings derived from camera calibration and camera tokens, then processes them with factorized self, temporal, and view attention. The work it does is to provide a shared latent in which surface-level depth, instance-level 3D boxes, and volume-level occupancy can be decoded from the same features.
What would settle it
Train the same GeoUP architecture with a randomly initialized VGGT-12 block and no reconstruction pretraining while keeping everything else identical, then compare detection mAP and occupancy mIoU on the nuScenes validation set: if the gap over the DINOv2-L baseline collapses, the claimed geometry transfer from reconstruction pretraining is not the driving factor.
Extended reading notes
Core claim
GeoUP claims that a reconstruction-oriented latent from VGGT, adapted to calibrated streaming multi-camera driving scenes, yields a shared representation that simultaneously supports metric depth estimation, 3D object detection, and semantic occupancy prediction at state-of-the-art accuracy. The adaptation consists of factorizing cross-image attention into self, temporal, and view attention, plus injecting calibration-aware Plücker raymap encodings for metric scale. The paper reports that this geometry-grounded latent outperforms prior task-specific approaches on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD, and that multi-task joint training across datasets further improves all tasks.
Load-bearing premise
The paper assumes that a 12-block subset of VGGT, initialized from the reconstruction-pretrained checkpoint and fine-tuned with depth and camera losses, actually transfers reconstruction-oriented geometry to a metric driving perception latent, rather than the gains coming mostly from its larger ViT-L backbone, multi-task training, or longer schedules.
Editorial extensions
If this is right
- Camera-only 3D detection, occupancy, and depth could be unified under one pretrained geometry backbone instead of separate task-specific modules.
- Joint training on datasets with heterogeneous annotations becomes a way to strengthen the shared 3D latent, since supervision for any task feeds the same representation.
- End-to-end planners can inherit a geometry-grounded perception backbone and improve their closed-loop safety metrics without retraining the planner.
- The geometry-grounded latent preserves reconstruction capability, so it can produce consistent point maps from predicted depth on driving scenes.
Reading between the lines
- A testable extension is to probe whether the latent's geometry comes from the VGGT pretraining objective or simply from multi-view temporal data at training time, by training VGGT-12 without the reconstruction checkpoint on the same tasks.
- The factorization into self/temporal/view attention suggests that deploying GeoUP in a streaming setting could cache and recycle attention states to amortize the cost of the slow multi-frame backbone.
- Because the paper reports detection results only on nuScenes-style category spaces, the cross-dataset sharing of the geometry latent may not yet demonstrate open-vocabulary or unseen-category detection, which would be a natural next test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GeoUP, a unified camera-only 3D perception framework that adapts the reconstruction-pretrained VGGT transformer to calibrated streaming multi-camera driving scenes. It factorizes cross-image attention into self, temporal, and view attention, injects calibration-aware Plücker raymap embeddings, and decodes the resulting shared latent with task-specific heads for metric depth estimation, 3D object detection, and semantic occupancy prediction. The authors report state-of-the-art results on nuScenes, Argoverse 2, Waymo, Occ3D-nuScenes, KITTI, and DDAD, and a NAVSIMv2 planning-transfer experiment, with component ablations over attention factorization, temporal context, backbone initialization, and multi-task learning.
Significance. If the central claim holds, the paper would be a strong demonstration that reconstruction-oriented latents from visual geometry foundation models can serve as a unified backbone for multiple driving perception tasks, rather than requiring task-specific geometry modules on top of recognition backbones. The paper contains useful independent evidence for this view, particularly the VGGT-initialization-versus-random-initialization comparison in Table 9 and the component ablations in Tables 7 and 10. The multi-dataset results in Table 13 and the planning-transfer experiment add breadth. However, the headline claims currently exceed the evidence: the multi-dataset gains are confounded by changes in training schedule, batch size, and optimizer, and the depth comparison omits standard task-specific metric-depth baselines. The paper is a solid systems contribution whose main claims require additional controlled experiments to be fully supported.
major comments (4)
- [§4.2, Tables 1-5, Table 13] The multi-dataset model GeoUP† is trained with twice the epochs, batch size 64 versus 16, and the Muon optimizer instead of AdamW relative to single-dataset GeoUP. The consistent gains attributed to heterogeneous supervision and multi-dataset training are therefore confounded with additional compute and a stronger optimizer. Please provide a same-schedule control, such as multi-dataset training with batch size 16 and AdamW or single-dataset training with batch size 64 and Muon, so that the effect of multi-dataset supervision is isolated; without this control, the conclusion that heterogeneous supervision strengthens the geometry-grounded representation is not established.
- [Table 5, §4.3] The depth comparison considers only reconstruction-oriented VGGT-family baselines (StreamVGGT, VGGT, MapAnything, DVGT) and omits task-specific metric depth models such as Metric3D or metric Depth Anything variants. The claim of state-of-the-art depth estimation is therefore not supported. Please add standard metric-depth baselines on KITTI and DDAD, or revise the claim to state superiority over visual-geometry reconstruction models only.
- [Table 9, §4.4] The backbone ablation does not isolate the VGGT initialization for the full GeoUP architecture. The DINOv2-T row uses randomly initialized VGGT-style blocks without the driving-oriented adaptations, so the comparison between GeoUP and DINOv2-T conflates architecture, adaptation, and initialization. Please add a control consisting of the full GeoUP architecture (self/temporal/view attention plus raymap injection) with random initialization, trained under the same schedule; this is the missing experiment that directly tests whether geometry pretraining, rather than the larger ViT-L backbone and multi-task training, is responsible for the reported gains.
- [Table 8, §4.4] In the frame-count ablation, the 1-frame row shows a large gap between current-only heads and temporal heads (48.7 vs 55.1 mAP), although with a single input frame the two settings should coincide. Please clarify what the temporal heads do with one frame or correct the table; as printed, this inconsistency prevents a clean interpretation of the temporal-modeling contribution.
minor comments (7)
- [Table 5, §4.3] On KITTI, GeoUP† (0.075 Abs Rel) is worse than single-dataset GeoUP (0.072), which undercuts the phrase "consistent gains" when applied to depth; please discuss this exception or explain why multi-dataset training slightly degrades KITTI depth.
- [Table 7 vs Table 9] The "Global" row in Table 7 (53.8 mAP, 61.7 NDS) appears to describe the same VGGT-12 global-attention configuration as the VGGT-12 row in Table 9 (54.6 mAP, 62.0 NDS), but the numbers differ; please clarify whether these are different settings or explain the discrepancy.
- [Section 3.3, Appendix A] The depth loss is described as combining regression and gradient-based regularization, but Eq. (10) gives equal weights of 1.0 for both terms; please state whether these weights match the main experiments and define the loss weights precisely for reproducibility.
- [Table 9] The row name "DINOv2-T" is confusing; please specify that it denotes a DINOv2 encoder augmented with 12 randomly initialized VGGT-style transformer blocks, and clarify which DINOv2 encoder variant is used.
- [§4.2] The construction of VGGT-12 by selecting every other layer from the pretrained VGGT checkpoint is presented without justification or sensitivity analysis; please report whether other layer subsets give similar results, or note this as a limitation.
- [Appendix A, §3.2] The unified depth scale of 90 m used to normalize depth targets across datasets is mentioned only in Appendix A; please state it in Section 3.2 or in the implementation details for reproducibility.
- [Table 6] The EPDMS results are reported as "original/corrected" in Table 6, and the text refers to "original and corrected evaluators" without defining which entry corresponds to which; please state the correspondence explicitly.
Circularity Check
No significant circularity: GeoUP is an empirical system validated on external public benchmarks, with independent ablation evidence for the geometry-grounding claim.
full rationale
GeoUP's central claim is that a VGGT-initialized, calibration-aware latent improves three driving perception tasks. This is an empirical claim, not a derivation that reduces to its inputs. The geometry grounding is supported by Table 9, which holds architecture and input configuration fixed and shows that VGGT-12 initialization outperforms DINOv2-T (randomly initialized VGGT-style blocks) by 2.5 mAP/1.7 NDS and 1.7 mIoU/0.8 RayIoU; this ablation is self-contained and does not presuppose the target result. The raymap and camera-token injections (Eqs. 1-2) encode camera intrinsics and poses, which are inputs rather than fitted predictions, and the depth, detection, and occupancy heads are supervised by external dataset labels that are not derived from the backbone's own outputs. The first-stage depth+camera fine-tuning (Eq. 8) followed by multi-task training (Eq. 9) is standard adaptation, not a fitted quantity being renamed as a prediction. The paper's cited backbone VGGT and task heads RayDN and OPUS-V2 are external works with no author overlap, and no load-bearing uniqueness theorem or ansatz is imported through self-citation. The skeptical concerns about longer schedules, larger batches, and the Muon optimizer in multi-dataset training, and about missing task-specific metric-depth baselines in Table 5, are experimental-validity or comparison-adequacy concerns rather than circularity. The only self-citation in the reference list ([30], involving authors Han Li, Zehao Huang, Naiyan Wang, and Si Liu) is not used as evidence for any central premise. Thus the paper's derivation chain is self-contained against external benchmarks, and the appropriate circularity finding is none.
Assumptions & free parameters
free parameters (5)
- Unified depth scale =
90 m
- Dataset point-cloud ranges =
51.2 m / 74.88 m / 152.4 m
- Dataset sampling ratio =
8:8:9:3:1
- Loss weights lambda_dep and lambda_cam =
0.1 each
- Temporal window length =
4 frames
assumptions (5)
- domain assumption The selected 12-block subset of VGGT retains enough reconstruction-oriented geometry that, after first-stage depth and camera fine-tuning, it forms a metric multi-view driving latent.
- domain assumption Camera intrinsics and camera-to-reference poses are accurately known and calibration does not drift.
- domain assumption DINOv2 patch features retain semantic discriminability under driving domain shift.
- standard math Plucker ray coordinates are a valid 6D line parametrization.
- domain assumption The nuScenes-style LiDAR coordinate convention unifies 3D box annotations across datasets.
Cite this review
Pith. "Pith review of Geometry-Grounded Unified 3D Perception for Autonomous Driving." pith.science (2026). https://pith.science/paper/7WXXNOR5
@misc{pith2026260813147,
author = {Pith},
title = {Pith review of: Geometry-Grounded Unified 3D Perception for Autonomous Driving},
year = {2026},
howpublished = {\url{https://pith.science/paper/7WXXNOR5}},
note = {Machine review of arXiv:2608.13147}
}
read the original abstract
Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021
arXiv 2021
-
[2]
Must3r: Multi-view network for stereo 3d re- construction
Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy. Must3r: Multi-view network for stereo 3d re- construction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1050–1060, 2025
2025
-
[3]
nuscenes: A mul- timodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020
2020
-
[4]
Pseudo-simulation for autonomous driv- ing
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Pseudo-simulation for autonomous driv- ing. InConference on Robot Learning (CoRL), 2025
work page 2025
-
[5]
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kir- illov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuro- pean conference on computer vision, pages 213–229, 2020
work page 2020
-
[6]
Emerging properties in self-supervised vision trans- formers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self-supervised vision trans- formers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021
work page 2021
-
[7]
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational confer- ence on machine learning, pages 1597–1607, 2020
work page 2020
-
[8]
Densebev: Transforming bev grid cells into 3d objects
Marius Dähling, Sebastian Krebs, and J Marius Zöllner. Densebev: Transforming bev grid cells into 3d objects. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2370–2379, 2026
work page 2026
Show all 82 references
-
[9]
Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing Systems, 37:28706–28719, 2024
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing ...
2024
-
[10]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009
2009
-
[11]
An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXi...
2010 arXiv
-
[12]
Centernet: Keypoint triplets for object detection
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. InProceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019
2019
-
[13]
Eva: Exploring the limits of masked visual rep- resentation learning at scale
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual rep- resentation learning at scale. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pa...
2023
-
[14]
Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149: 105171, 2024
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149: 105171, 2024
2024
-
[15]
Vision meets robotics: The kitti dataset.The international journal of robotics research, 32(11): 1231–1237, 2013
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset.The international journal of robotics research, 32(11): 1231–1237, 2013
2013
-
[16]
Digging into self-supervised monocular depth estimation
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828–3838, 2019
2019
-
[17]
3d packing for self-supervised monocular depth estimation
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2485–2494, 2020
2020
-
[18]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[19]
Momentum con- trast for unsupervised visual representation learning
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum con- trast for unsupervised visual representation learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020
2020
-
[20]
Masked autoencoders are scalable vision learners
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022
2022
-
[21]
Bevdet: High- performance multi-camera 3d object detection in bird-eye-view.arXiv preprint arXiv:2112.11790, 2021
Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High- performance multi-camera 3d object detection in bird-eye-view.arXiv preprint arXiv:2112.11790, 2021. XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION17
2021 arXiv
-
[22]
Tri- perspective view for vision-based 3d semantic occupancy prediction
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri- perspective view for vision-based 3d semantic occupancy prediction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9223– 9232, 2023
2023
-
[23]
Ropetr: Improving temporal camera-only 3d detection by integrating enhanced rotary position embedding.arXiv preprint arXiv:2504.12643, 2025
Hang Ji, Tao Ni, Xufeng Huang, Zhan Shi, Tao Luo, Xin Zhan, and Junbo Chen. Ropetr: Improving temporal camera-only 3d detection by integrating enhanced rotary position embedding.arXiv preprint arXiv:2504.12643, 2025
2025 arXiv
-
[24]
Drivevggt: Visual geometry transformer for autonomous driving.arXiv preprint arXiv:2511.22264, 2025
Xiaosong Jia, Yanhao Liu, Junqi You, Renqiu Xia, Yu Hong, and Junchi Yan. Drivevggt: Visual geometry transformer for autonomous driving.arXiv preprint arXiv:2511.22264, 2025
2025
-
[25]
Far3d: Expanding the horizon for surround-view 3d object detection
Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xiangyu Zhang. Far3d: Expanding the horizon for surround-view 3d object detection. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 2561–2569, 2024
2024
-
[26]
Muon: An optimizer for hidden layers in neural networks, 2024
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon, 6(3):4, 2024
2024
-
[27]
Mapanything: Universal feed-forward metric 3d reconstruction.arXiv preprint arXiv:2509.13414, 2025
Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, et al. Mapanything: Universal feed-forward metric 3d reconstruction.arXiv preprint arXiv:2509.13414, 2025
2025 arXiv
-
[28]
An energy and gpu-computation efficient backbone network for real-time object detection
Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpu-computation efficient backbone network for real-time object detection. InProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion workshops, pages 0–0, 2019
2019
-
[29]
Grounding image matching in 3d with mast3r
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. InEuropean Conference on Computer Vision, pages 71–91, 2024
2024
-
[30]
En- hancing 3d lane detection and topology reasoning with 2d lane priors.arXiv preprint arXiv:2406.03105, 2024
Han Li, Zehao Huang, Zitian Wang, Wenge Rong, Naiyan Wang, and Si Liu. En- hancing 3d lane detection and topology reasoning with 2d lane priors.arXiv preprint arXiv:2406.03105, 2024
2024 arXiv
-
[31]
Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.Advances in neural information processing systems, 33: 21002–21012, 2020
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.Advances in neural information processing systems, 33: 21002–21012, 2020
2020
-
[32]
Bevdepth: Acquisition of reliable depth for multi-view 3d object detection
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. InProceedings of the AAAI conference on artificial intelligence, volume 37, pages 1477–1485, 2023
2023
-
[33]
Fb-occ: 3d occupancy prediction based on forward-backward view transformation.arXiv preprint arXiv:2307.01492, 2023
Zhiqi Li, Zhiding Yu, David Austin, Mingsheng Fang, Shiyi Lan, Jan Kautz, and Jose M Alvarez. Fb-occ: 3d occupancy prediction based on forward-backward view transformation.arXiv preprint arXiv:2307.01492, 2023. 18XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION
2023 arXiv
-
[34]
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird’s-eye-view representation from lidar-camera via spatiotemporal transformers.IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3):2020–2036, 2024
2020
-
[35]
Focal loss for dense object detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. InProceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017
2017
-
[36]
Ray denoising: Depth-aware hard negative sampling for multi-view 3d object detection
Feng Liu, Tengteng Huang, Qianjing Zhang, Haotian Yao, Chi Zhang, Fang Wan, Qix- iang Ye, and Yanzhao Zhou. Ray denoising: Depth-aware hard negative sampling for multi-view 3d object detection. InEuropean Conference on Computer Vision, pages 200–217. Springer, 2024
2024
-
[37]
Fully sparse 3d occupancy prediction
Haisong Liu, Yang Chen, Haiguang Wang, Zetong Yang, Tianyu Li, Jia Zeng, Li Chen, Hongyang Li, and Limin Wang. Fully sparse 3d occupancy prediction. InEuropean Conference on Computer Vision, pages 54–71. Springer, 2024
2024
-
[38]
Petr: Position embedding transformation for multi-view 3d object detection
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. InEuropean Conference on Com- puter Vision, pages 531–548, 2022
2022
-
[39]
Petrv2: A unified framework for 3d perception from multi-camera images
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3262–3272, 2023
2023
-
[40]
Swin transformer: Hierarchical vision transformer using shifted win- dows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted win- dows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021
2021
-
[41]
Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In2023 IEEE international conference on robotics and automation (ICRA), pages 2774–2781. IEEE, 2023
2023
-
[42]
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022
2022
-
[43]
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016
2016 arXiv
-
[44]
Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017
2017 arXiv
-
[45]
Ovggt: O(1) constant-cost streaming visual geometry transformer.arXiv preprint arXiv:2603.05959, 2026
Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu, Sin-Ye Jhong, Wen-Huang Cheng, and Yung- Yao Chen. Ovggt: O(1) constant-cost streaming visual geometry transformer.arXiv preprint arXiv:2603.05959, 2026. XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION19
2026 arXiv
-
[46]
Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023
2023 arXiv
-
[47]
Is pseudo- lidar needed for monocular 3d object detection? InProceedings of the IEEE/CVF international conference on computer vision, pages 3142–3152, 2021
Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo- lidar needed for monocular 3d object detection? InProceedings of the IEEE/CVF international conference on computer vision, pages 3142–3152, 2021
2021
-
[48]
Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. InEuropean conference on computer vision, 2020
2020
-
[49]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational confer- ence on machine learning, p...
2021
-
[50]
Vision transformers for dense prediction
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021
2021
-
[51]
Superglue: Learning feature matching with graph neural networks
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938– 4947, 2020
2020
-
[52]
Structure-from-motion revisited
Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016
2016
-
[53]
Fastvggt: Training-free acceleration of visual geometry transformer
You Shen, Zhipeng Zhang, Yansong Qu, Xiawu Zheng, Jiayi Ji, Shengchuan Zhang, and Liujuan Cao. Fastvggt: Training-free acceleration of visual geometry transformer. arXiv preprint arXiv:2509.02560, 2025
2025 arXiv
-
[54]
3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection
Changyong Shu, Jiajun Deng, Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3580–3589, 2023
2023
-
[55]
Dinov3.arXiv preprint arXiv:2508.10104, 2025
Oriane Siméoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025
2025 arXiv
-
[56]
Light field networks: Neural scene representations with single-evaluation render- ing.Advances in Neural Information Processing Systems, 34:19313–19325, 2021
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Du- rand. Light field networks: Neural scene representations with single-evaluation render- ing.Advances in Neural Information Processing Systems, 34:19313–19325, 2021
2021
-
[57]
Loftr: Detector-free local feature matching with transformers
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021. 20XUet al.: GEOMETRY -GROUNDED UNIFIED ...
2021
-
[58]
Scalability in perception for autonomous driving: Waymo open dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InProceedings of the IEEE/CVF conference on computer vis...
2020
-
[59]
Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving.Advances in Neural Information Processing Systems, 36:64318– 64330, 2023
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving.Advances in Neural Information Processing Systems, 36:64318– 64330, 2023
2023
-
[60]
Patchmatchnet: Learned multi-view patchmatch stereo
Fangjinhua Wang, Silvano Galliani, Christoph V ogel, Pablo Speciale, and Marc Polle- feys. Patchmatchnet: Learned multi-view patchmatch stereo. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14194– 14203, 2021
2021
-
[61]
Opus: occupancy prediction using a sparse set
Jiabao Wang, Zhaojiang Liu, Qiang Meng, Liujiang Yan, Ke Wang, Jie Yang, Wei Liu, Qibin Hou, and Ming-Ming Cheng. Opus: occupancy prediction using a sparse set. Advances in Neural Information Processing Systems, 37:119861–119885, 2024
2024
-
[62]
Vggsfm: Vi- sual geometry grounded deep structure from motion
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Vi- sual geometry grounded deep structure from motion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21686–21697, 2024
2024
-
[63]
Vggt: Visual geometry grounded transformer
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5294– 5306, 2025
2025
-
[64]
Exploring object-centric temporal modeling for efficient multi-view 3d object detection
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. InPro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621– 3631, 2023
2023
-
[65]
Dust3r: Geometric 3d vision made easy
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697–20709, 2024
2024
-
[66]
Fcos3d: Fully convolutional one-stage monocular 3d object detection
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. InProceedings of the IEEE/CVF interna- tional conference on computer vision, pages 913–922, 2021
2021
-
[67]
Mv-fcos3d++: Multi-view camera-only 4d object detection with pretrained monocular backbones
Tai Wang, Qing Lian, Chenming Zhu, Xinge Zhu, and Wenwei Zhang. Mv-fcos3d++: Multi-view camera-only 4d object detection with pretrained monocular backbones. arXiv preprint arXiv:2207.12716, 2022
2022 arXiv
-
[68]
XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION21
Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He.π 3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025. XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION21
2025 arXiv
-
[69]
Detr3d: 3d object detection from multi-view images via 3d-to-2d queries
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. InConference on robot learning, pages 180–191, 2022
2022
-
[70]
Object as query: Lifting any 2d object detector to 3d detection
Zitian Wang, Zehao Huang, Jiahui Fu, Naiyan Wang, and Si Liu. Object as query: Lifting any 2d object detector to 3d detection. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 3791–3800, 2023
2023
-
[71]
Sur- roundocc: Multi-camera 3d occupancy prediction for autonomous driving
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Sur- roundocc: Multi-camera 3d occupancy prediction for autonomous driving. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 21729– 21740, 2023
2023
-
[72]
Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:2301.00493, 2023
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Sid- dhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:23...
2023 arXiv
-
[73]
Bevformer v2: Adapting mod- ern image backbones to bird’s-eye-view recognition via perspective supervision
Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting mod- ern image backbones to bird’s-eye-view recognition via perspective supervision. In Proceedings of the IEEE/CVF conference...
2023
-
[74]
Depth anything: Unleashing the power of large-scale unlabeled data
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10371–10381. IEEE, 2024
2024
-
[75]
Drivesuprim: Towards precise trajectory selection for end-to-end plan- ning
Wenhao Yao, Zhenxin Li, Shiyi Lan, Zi Wang, Xinglong Sun, Jose M Alvarez, and Zuxuan Wu. Drivesuprim: Towards precise trajectory selection for end-to-end plan- ning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11910–11918, 2026
2026
-
[76]
Mvsnet: Depth inference for unstructured multi-view stereo
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. InProceedings of the European conference on computer vision (ECCV), pages 767–783, 2018
2018
-
[77]
Infinitevggt: Visual geometry grounded transformer for endless streams.arXiv preprint arXiv:2601.02281, 2026
Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, and Zhipeng Zhang. Infinitevggt: Visual geometry grounded transformer for endless streams.arXiv preprint arXiv:2601.02281, 2026
2026
-
[78]
Sigmoid loss for language image pre-training
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international con- ference on computer vision, pages 11975–11986, 2023
2023
-
[79]
Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction
Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 9433–9443, 2023. 22XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION
2023
-
[80]
Free3d: Consistent novel view synthesis without 3d representation
Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d representation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9720–9731, 2024
2024
-
[81]
Streaming 4d visual geometry transformer.arXiv preprint arXiv:2507.11539, 2025
Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu. Streaming 4d visual geometry transformer.arXiv preprint arXiv:2507.11539, 2025
2025 arXiv
-
[82]
Dvgt: Driving visual geometry transformer
Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Shengyin Jiang, Long Chen, Zhi-Xin Yang, and Jiwen Lu. Dvgt: Driving visual geometry transformer. arXiv preprint arXiv:2512.16919, 2025. XUet al.: GEOMETRY -GROUNDED UNIFIED 3D PERCEPTION1 Appendix A Additional Datas...
2025 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.