Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Frozen DINOv2 and Metric3Dv2 lift BEV vehicle segmentation from 33.0 to 41.9 IoU.

desk verdict Empirical swap of frozen DINOv2/Metric3Dv2 into BEV pipelines shows genuine gains, but single-seed runs and an unverified zero-shot claim for Metric3Dv2 make the headline numbers fragile. read the letter →

arxiv 2501.08118 v1 pith:GM3SHQLE submitted 2025-01-14 cs.CV

classification cs.CV
keywords birdseyeviewperceptionvehiclesegmentationfoundationmodelsDINOv2Metric3Dv2PseudoLiDARnuScenesfrozenencoders
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf, frozen foundation models can be dropped into existing bird's-eye-view (BEV) perception pipelines to reduce training data and beat the original models. By replacing Lift-Splat-Shoot's EfficientNet encoder with a frozen DINOv2 feature extractor and its learned depth module with Metric3Dv2 metric depth, the authors report vehicle segmentation IoU rising from 33.0 to 41.9 on nuScenes while using half the training data and far fewer iterations. In Simple-BEV, they convert Metric3Dv2 depth images into a PseudoLiDAR point cloud and show it beats the camera-only model by roughly 3 IoU at full image resolution and 6.4 IoU at half resolution. The paper's point is that careful integration of frozen foundation models can modernize old architectures cheaply, with the caveat that the size of the depth model matters more than the size of the feature extractor.

What carries the argument

The load-bearing mechanism is the frozen foundation-model pair inserted into Lift-Splat-Shoot: DINOv2 supplies 14x14-patch image features with fine granularity, and Metric3Dv2 supplies per-pixel metric depth in meters. To connect Metric3Dv2 to LSS's 41-bin depth distribution, the paper pools continuous depth values into non-overlapping 16x16 patches and assigns each pixel's depth to the corresponding meter bin, producing a tensor of shape 41 by H/16 by W/16; a same-convolutional layer with batch normalization and ReLU stabilizes it. For Simple-BEV, the mechanism is PseudoLiDAR: Metric3Dv2 depth images are unprojected into 3D point clouds using camera intrinsics and extrinsics, then fed through Simple-BEV's standard point-cloud occupancy pathway while retaining bilinear-sampling splatting for image features.

What would settle it

Train the same top configurations (DINOv2 Giant plus Metric3Dv2 Giant LSS, and Camera plus PseudoLiDAR Simple-BEV) at least five times with different random seeds and report the mean and standard deviation of peak IoU; if the standard deviation exceeds the reported 3 to 6 IoU gaps, or if the baseline catches up within noise, the central claim fails.

Watch

Extended reading notes

Core claim

The core discovery is that replacing the learned components of an established BEV pipeline with frozen foundation models is not only compatible but beneficial. In Lift-Splat-Shoot, substituting EfficientNet with DINOv2 image features and the learned depth distribution with Metric3Dv2's metric depth raised peak vehicle segmentation IoU from 33.0 to 41.9, and the same configuration trained on half the nuScenes training split reached 40.4 IoU, outperforming the original model trained on the full dataset. Ablations show the depth model's scale drives most of the gain: swapping Metric3Dv2 from Small to Giant adds roughly 5.8 IoU, while DINOv2 scale adds about 1.8 IoU. In Simple-BEV, a PseudoLiDAR point cloud generated from Metric3Dv2 depth and projected with camera intrinsics and extrinsics yields 50.7 IoU versus 47.4 for camera-only at 448x800 input, and a larger gain at reduced input resolution, showing the depth signal substitutes for what the camera model failed to infer.

Load-bearing premise

The load-bearing premise is that the reported IoU gaps are real and not noise: every configuration is trained once and scored at its peak iteration, so if run-to-run variance is comparable to gaps like the +3.0 or +6.4 PseudoLiDAR improvements, the conclusions could reverse under a different seed.

Editorial extensions

If this is right

  • Frozen DINOv2 plus Metric3Dv2 can replace EfficientNet and learned depth in Lift-Splat-Shoot to gain about 8.9 IoU over the full-data baseline, and still beat that baseline by about 7.4 IoU when trained on half the data.
  • Metric3Dv2 scale matters more than DINOv2 scale for LSS performance, with the Giant depth model adding roughly 5.8 IoU over the Small one while DINOv2 scale adds about 1.8 IoU.
  • PseudoLiDAR from Metric3Dv2 depth is a viable cheap substitute for actual LiDAR in Simple-BEV, improving camera-only by about 3 IoU at full resolution and 6.4 IoU at half resolution.
  • Using frozen foundation models dramatically cuts training iterations: the modified LSS peaks at 55k iterations versus 300k+ for the original and outperforms the original after about 5k iterations.
  • The approach suggests camera-only BEV perception can reduce dependence on LiDAR hardware, lowering sensor cost while preserving much of the multimodal performance gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the depth-pooling conversion is generic: any per-pixel monocular depth source could replace Metric3Dv2 in LSS, making this a reusable test harness for future metric-depth models.
  • An untested corollary is that PseudoLiDAR will help most in low-resolution or degraded-camera settings, since the paper's largest relative gain appears when the camera input resolution is halved.
  • We note that the paper's single-run protocol leaves the exact magnitudes uncertain; a natural extension is multi-seed training of the top configurations to establish whether the 3 to 6 IoU gaps exceed run-to-run variance.
  • A further inference is that combining PseudoLiDAR with radar rather than replacing either modality could yield complementary depth cues, though the paper does not test this combination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript reports an empirical study in which frozen pretrained foundation models are substituted into two BEV perception pipelines. For Lift-Splat-Shoot, the EfficientNet feature/depth head is replaced by frozen DINOv2 features and Metric3Dv2 metric depth, and the authors report that the best configuration improves vehicle segmentation IoU on nuScenes from 33.0 to 41.9 when trained on the full dataset and from 29.1 to 40.4 when trained on half the data, with the latter described as surpassing the full-data baseline by 7.4 IoU (Table 4). For Simple-BEV, Metric3Dv2 depth images are converted into a PseudoLiDAR point cloud and injected alongside camera features, giving +3.0 IoU over the Camera-only model at 448x800 input and +6.3 IoU at 224x400 input (Tables 5 and 6). The paper also includes ablations over DINOv2 and Metric3Dv2 model sizes. The central claim is that frozen foundation models can be dropped into existing BEV architectures to improve accuracy while reducing required training data and iterations.

Significance. If the reported results are robust, the paper would provide a practically valuable demonstration that frozen general-purpose vision models can replace learned task-specific encoders and depth estimators in BEV perception, with implications for data efficiency and sensor cost. The integration recipe is described clearly: the depth-to-41-bin pooling for LSS is a concrete and reproducible design choice, and the PseudoLiDAR construction for Simple-BEV is a plausible alternative to real LiDAR. The ablations over model size are also a useful contribution. However, the paper's load-bearing evidence is currently thin: every configuration is evaluated from a single training run using a peak-IoU selection rule, the headline comparisons mix several variables, and the zero-shot transfer claim depends on an unverified assumption about Metric3Dv2's pretraining corpus. These issues do not invalidate the approach, but they need to be addressed before the accuracy gains can be considered established.

major comments (5)
  1. [§Results: Comparison with Original Model, Table 4] The headline 7.4 IoU gain compares the half-data DINOv2+Metric3Dv2 Giant model (40.4) with the full-data EfficientNet baseline (33.0), so architecture, pretraining, and training-data volume all vary simultaneously. This does not support the statement that foundation models reduce required training data while surpassing baseline performance. Please report the same-architecture full-data result alongside the half-data result, and provide a matched comparison at fixed iteration or wall-clock budgets, so that the effect of data volume is isolated from the effect of model choice.
  2. [§Results: Largeness of Foundation Models, Tables 2–6] All reported numbers appear to be single-run peak IoU values selected across training iterations, with no error bars, no multiple seeds, and no explicit checkpoint-selection rule. Given the typical run-to-run variation in BEV segmentation training, differences such as the +1.8 IoU attributed to DINOv2 size or the +3.0 IoU attributed to PseudoLiDAR could be within noise. Please train each key configuration with at least three seeds and report mean and standard deviation, and specify whether the reported IoU is the best validation score during training or a fixed final checkpoint.
  3. [§Metric3Dv2 and §Metric Depth Distribution] The paper interprets the gains in Tables 4 and 5 as evidence of zero-shot transfer from a frozen generalist depth model, but it never verifies that Metric3Dv2's pretraining corpus excludes nuScenes. Metric3Dv2 is trained on about 8 million images from many datasets, and driving datasets with metric depth are common in such pretraining. If nuScenes appears in that corpus, the improvements are partly memorization of the evaluation domain rather than transfer, and the comparison with an ImageNet-only EfficientNet baseline is asymmetric. Please either provide a list of Metric3Dv2's training datasets with confirmation that nuScenes is excluded, or rerun the key comparisons with a depth model verifiably not trained on nuScenes; an additional cross-dataset evaluation would further strengthen the transfer claim.
  4. [§Results: Comparison with Original Model, Table 4] The reported iteration counts differ wildly across configurations (e.g., 55k for DINOv2+Metric3Dv2 Giant versus 300k for the baseline). Reporting peak IoU at such different training durations makes it difficult to tell whether the gain comes from the frozen features, the depth signal, or simply from a different effective training schedule. Please report learning curves or at least IoU at matched iteration counts, and state the total compute per configuration, since the Giant models are reported to be roughly 7.5x slower per iteration.
  5. [§Construction of a PseudoLiDAR Point Cloud, Tables 5–6] The comparison for Simple-BEV is Camera+PseudoLiDAR versus Camera-only, but the PseudoLiDAR cloud is generated from the same camera image using the same intrinsics/extrinsics. The +3.0 IoU therefore quantifies the value of adding this depth-derived representation, yet the paper does not compare against an alternative way of injecting the same depth signal, such as a learnable depth branch or depth-based splatting with matched parameters and compute. The paper cites Simple-BEV's finding that depth splatting loses 3 IoU, but does not implement that variant. Adding such a control would clarify whether the gain is specific to the PseudoLiDAR representation or simply to providing metric depth information.
minor comments (6)
  1. [Throughout] Several cross-references are broken: the text says "described later in Section ." and "detailed in Section ." without numbers, and there are rendering artifacts such as "F or" in the abstract.
  2. [References] Reference [7] is a DINOv2 application paper for parasite classification, not the original DINOv2 model paper. The authors should cite Oquab et al., "DINOv2: Learning Robust Visual Features without Supervision" (2023), for the model description and pretraining details.
  3. [Table 2] The row "Giant Giant 41.9 55k" appears twice; this appears to be a typo, and one copy should be removed.
  4. [Tables 5–6 and Figure 4] The PseudoLiDAR depth image resolution is inconsistently reported as (112,200) in Table 5 and Figure 4 but as 112x224 in the text and abstract. Please make the dimensions consistent.
  5. [§Results: Largeness of Foundation Models] The names "Small", "Base", "Large", and "Giant" for DINOv2 and Metric3Dv2 are not formally defined. Please specify the exact model variants (e.g., ViT-S, ViT-B, ViT-L, ViT-g) and patch sizes so the experiments can be reproduced.
  6. [§Results: Comparison with Original Model] There is a typo in "altered LLS architecture" where "LLS" should be "LSS", and the caption of Table 4 says "LSS" while the text uses both spellings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical integration of externally pretrained foundation models into BEV pipelines, with all load-bearing comparisons measured against external baselines.

full rationale

The paper makes no derivation-style claim that could reduce to its inputs. Its central results are measured IoU values on nuScenes validation for modified LSS and Simple-BEV architectures, compared against baseline models trained and evaluated by the same protocol. The inputs are frozen, externally released models (DINOv2 and Metric3Dv2), and the paper never fits a parameter to the target IoU and then re-presents it as a prediction. The depth-to-distribution conversion in the LSS modification is a fixed pooling operation over Metric3Dv2 output, not a learned or fitted quantity. The PseudoLiDAR point cloud is generated by unprojecting Metric3Dv2 depth using known camera intrinsics/extrinsics, then fed through the standard Simple-BEV pipeline; its +3 IoU over Camera-only is an empirical measurement, not an identity. No load-bearing argument relies on self-citations: the cited sources for LSS, Simple-BEV, DINOv2, and Metric3Dv2 are external prior work, and no uniqueness theorem or prior result by the present authors is invoked to forbid alternatives. The only substantive caveat is the skeptic's concern that Metric3Dv2 may have been pretrained on nuScenes or similar driving data, which would weaken the attribution of gains to zero-shot transfer. That is a data-contamination or external-validity question, not circular reasoning. Under the stated rules, such an external-attribution concern does not constitute circularity. The paper also openly acknowledges limitations such as runtime and that PseudoLiDAR does not match real LiDAR, further showing that the claims are qualified empirical results rather than forced conclusions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters; it relies entirely on pre-trained models and standard training. The main assumptions are the reliability of nuScenes labels, the transferability of frozen DINOv2 and Metric3Dv2 to driving data, and the validity of the depth pooling conversion. No new entities are invented.

assumptions (3)
  • domain assumption nuScenes annotations and the standard validation split are reliable indicators of BEV segmentation quality.
    All conclusions rest on the nuScenes benchmark as ground truth; the paper does not verify label quality.
  • domain assumption Frozen DINOv2 and Metric3Dv2 generalize to nuScenes camera images without fine-tuning.
    The core method assumes these pre-trained models produce useful features and metric depth on this driving dataset.
  • ad hoc to paper The depth pooling from Metric3Dv2 depth image to 41-bin distribution is a valid conversion for LSS.
    This conversion is invented for this work; its behavior and failure modes are not analyzed beyond the reported IoU.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2." pith.science (2026). https://pith.science/paper/GM3SHQLE

@misc{pith2026250108118,
  author       = {Pith},
  title        = {Pith review of: Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GM3SHQLE}},
  note         = {Machine review of arXiv:2501.08118}
}
read the original abstract

Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2's depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model.

Figures

Figures reproduced from arXiv: 2501.08118 by the authors.

Figure 1
Figure 1. Architecture Flow Diagram: Modified Lift-Splat ar￾chitecture with implementation of DINOv2 and Metric3Dv2 for image feature extraction and depth estimation primarily on the Lift component, which pertains to the acquisi￾tion of camera features and depth estimation. Here, replace the output of the EfficientNet encoder with DINOv2 image features and Metric3Dv2 depth to facilitate increased model performance, which will… view at source ↗
Figure 2
Figure 2. Architecture Flow Diagram: Simple BEV model for Camera+PseudoLiDAR. The Camera-only model excludes any point clouds, while Radar and LiDAR would substitute for Pseudo LiDAR The Simple BEV architecture allows us to implement radar or LiDAR data along with camera information which makes it a very versatile model. We extract features from each image and then perform bilinear sampling to lift these features into a 3D vo… view at source ↗
Figure 3
Figure 3. Metric3Dv2: Conversion from RGB image to depth image to depth distribution for compatibility with DINOv2 patch embeddings For the model architecture described in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Metric3Dv2 Point Cloud Sample: PseudoLiDAR point cloud extracted from a nuScenes sample at a depth image size of 112×200). RGB is coupled for visualisation purposes Some benefits of our Pseudo LiDAR point cloud can be ob￾served over LiDAR. Our PseudoLiDAR point cloud r…
Figure 5
Figure 5. Figure 5: Lift-Splat-Shoot: Comparison for full dataset and half dataset models. Vehicles are highlighted by coloured boxes for the purpose of analysis [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Simple-BEV Comparison: Camera only vs Cam￾era+Radar vs Camera+LiDAR vs Camera+Pseudo LiDAR. Vehi￾cles are highlighted by coloured boxes for the purpose of analysis Examining [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Eyes on the Road, Mind Beyond Vision: Context-Aware Multi-modal Enhanced Risk Anticipation

    cs.CE 2025-07 reject novelty 4.0 of 10

    CAMERA fuses video, text, and driver attention with a learned adaptive warning threshold to improve accident anticipation accuracy and lead time over published baselines.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [10]

    Robust Bird's Eye View Segmentation by Adapting DINOv2

    Barın, M. R., Aydemir, G., and G ¨uney, F. (2024). Robust Bird’s Eye View Segmentation by Adapting DINOv2. arXiv preprint arXiv:2409.10228

  2. [1]

    W., Fang, Z., Li, J., Ambrus, R., and Fragki- adaki, K

    Harley, A. W., Fang, Z., Li, J., Ambrus, R., and Fragki- adaki, K. (2023). Simple-BEV: What Really Matters for Multi-Sensor BEV Perception? In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2759–2765

  3. [2]

    H., V ora, S., Liong, V

    Caesar, H., Bankiti, V ., Lang, A. H., V ora, S., Liong, V . E., Xu, Q., Krishnan, A., Pan, Y ., Baldan, G., and Beijbom, O. (2020). nuScenes: A Multimodal Dataset for Autonomous Driving. In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 11618–11628

  4. [3]

    Xiong, Y ., Varadarajan, B., Wu, L., Xiang, X., Xiao, F., Zhu, C., Dai, X., Wang, D., Sun, F., Iandola, F., Kr- ishnamoorthi, R., and Chandra, V . (2024). EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16111–16121

  5. [4]

    C., Lo, W.- Y ., Dollar, P., and Girshick, R

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.- Y ., Dollar, P., and Girshick, R. (2023). Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026

  6. [5]

    R., and Koltun, V

    Bochkovskii, A., Delaunoy, A., Germain, H., Santos, M., Zhou, Y ., Richter, S. R., and Koltun, V . (2024). Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. arXiv preprint arXiv:2410.02073

  7. [6]

    Hu, M., Yin, W., Zhang, C., Cai, Z., Long, X., Chen, H., Wang, K., Yu, G., Shen, C., and Shen, S. (2024). Met- ric3Dv2 v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Esti- mation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–18

  8. [7]

    Pinetsuksai, N., Kittichai, V ., Jomtarak, R., Jaksukam, K., Tongloy, T., Boonsang, S., and Chuwongin, S. (2023). Development of Self-Supervised Learning with Dinov2- Distilled Models for Parasite Classification in Screening. In 2023 15th International Conference on Information Technology and Electrical Engineering (ICITEE), pages 323–328

Show all 10 references
  1. [8]

    and Fidler, S

    Philion, J. and Fidler, S. (2020). Lift-Splat-Shoot: Encod- ing Images from Arbitrary Camera Rigs by Implicitly Un- projecting to 3D. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV , pages 194–210, Berlin, Heide...

  2. [9]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. (2022). LoRA: Low-Rank Adap- tation of Large Language Models. In International Confer- ence on Learning Representations

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.