{"id":"4c5afd9e-4350-45fe-85ae-df91552956d0","arxiv_id":"2501.08118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frozen DINOv2 features and Metric3Dv2 depth improve Lift-Splat-Shoot BEV segmentation by up to 8.9 IoU and a Metric3Dv2 PseudoLiDAR cloud adds about 3 IoU to Simple-BEV camera-only.","lead":"The authors replace the feature and depth modules of two bird's-eye-view driving models with frozen foundation models, DINOv2 and Metric3Dv2, and report large IoU gains on the nuScenes dataset. The work suggests that large pre-trained models can cut training data needs in autonomous vehicle perception, though the results come without error bars or released code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot premise is unverified: if Metric3Dv2 was pretrained on nuScenes, the reported gains reflect memorization rather than frozen foundation-model transfer.","rationale":"The reader's weakest assumption was the lack of multiple seeds and variance reporting, which is a real concern for the small +3.0 IoU PseudoLiDAR gain. I agree that this compromises quantitative precision. However, the more load-bearing issue is the unexamined pretraining overlap of Metric3Dv2. The paper explicitly calls Metric3Dv2 'zero-shot' and uses it as a frozen depth source in both the LSS and Simple-BEV experiments. If Metric3Dv2 was pretrained on nuScenes, then the reported improvements are not evidence of generalizable frozen foundation models, but of a model that has already adapted to the target dataset. This would undermine the paper's central contribution: that foundation models can reduce the need for training data on a specific dataset. The manuscript provides no check for this, and the omission is a correctness risk rather than a stylistic one. I therefore recommend keeping a conditional verdict, but the condition should include verifying pretraining data disjointness, not only adding multiple seeds. The paper has merits: the runtime limitation is acknowledged, the PseudoLiDAR idea is clearly described, and the comparison tables are fairly organized. The concern I raise is falsifiable with public information about Metric3Dv2's training set, so it can be resolved without new experiments if the data list is clear. If the list is ambiguous, the proposed re-run with an alternative depth model settles it. I do not think the paper should be outright rejected because the empirical direction is plausible and the authors may simply have overlooked this verification; but the claim as written is not yet established.","tokens_in":8049,"tokens_out":4101,"duration_ms":44288,"concrete_test":"Obtain the official Metric3Dv2 training data list from the Metric3Dv2 paper, repository, or model card and check whether nuScenes or nuScenes-derived images are included. If nuScenes is present, rerun the Camera+PseudoLiDAR configuration from Table 5 using a depth model verifiably not trained on nuScenes (e.g., Depth Pro, if its training data excludes nuScenes) and compare the +3.0 IoU gain over Camera-only. If the gain disappears or shrinks materially, the central claim of zero-shot transfer is unsupported. If nuScenes is absent, the zero-shot assumption is likely safe, and the remaining risk is the single-run variance already noted by the reader.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that frozen foundation models (Metric3Dv2 for depth, DINOv2 for features) can be dropped into BEV pipelines to reduce training data and surpass baselines. This attribution rests on treating Metric3Dv2 as a zero-shot, generalist depth model. However, the manuscript nowhere verifies that Metric3Dv2's pretraining corpus excludes nuScenes. Metric3Dv2 is trained on roughly 8 million images spanning many datasets, and driving datasets with metric depth are common in such pretraining. If nuScenes (or nuScenes-derived images) appears in that corpus, then the +7.4 IoU improvement in Table 4 and the +3.0 IoU improvement in Table 5 are not evidence of transfer from a generalist; they are partly evidence that the depth model has already adapted to the exact sensor geometry, scene distribution, and annotation conventions of the test set. This is especially consequential because the comparison baseline (EfficientNet) is trained only on ImageNet, so the pretraining data overlap is asymmetric: the foundation model may have seen the evaluation domain, while the baseline has not. The concern is not about run-to-run noise; it invalidates the causal attribution even if every IoU number is perfectly reproducible. The paper should either demonstrate that Metric3Dv2's training set excludes nuScenes, or rerun the key comparisons with a depth model verifiably not trained on nuScenes, before claiming that frozen foundation models are responsible for the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical study in which frozen pretrained foundation models are substituted into two BEV perception pipelines. For Lift-Splat-Shoot, the EfficientNet feature/depth head is replaced by frozen DINOv2 features and Metric3Dv2 metric depth, and the authors report that the best configuration improves vehicle segmentation IoU on nuScenes from 33.0 to 41.9 when trained on the full dataset and from 29.1 to 40.4 when trained on half the data, with the latter described as surpassing the full-data baseline by 7.4 IoU (Table 4). For Simple-BEV, Metric3Dv2 depth images are converted into a PseudoLiDAR point cloud and injected alongside camera features, giving +3.0 IoU over the Camera-only model at 448x800 input and +6.3 IoU at 224x400 input (Tables 5 and 6). The paper also includes ablations over DINOv2 and Metric3Dv2 model sizes. The central claim is that frozen foundation models can be dropped into existing BEV architectures to improve accuracy while reducing required training data and iterations.","tokens_in":8261,"tokens_out":4958,"duration_ms":49397,"significance":"If the reported results are robust, the paper would provide a practically valuable demonstration that frozen general-purpose vision models can replace learned task-specific encoders and depth estimators in BEV perception, with implications for data efficiency and sensor cost. The integration recipe is described clearly: the depth-to-41-bin pooling for LSS is a concrete and reproducible design choice, and the PseudoLiDAR construction for Simple-BEV is a plausible alternative to real LiDAR. The ablations over model size are also a useful contribution. However, the paper's load-bearing evidence is currently thin: every configuration is evaluated from a single training run using a peak-IoU selection rule, the headline comparisons mix several variables, and the zero-shot transfer claim depends on an unverified assumption about Metric3Dv2's pretraining corpus. These issues do not invalidate the approach, but they need to be addressed before the accuracy gains can be considered established.","major_comments":[{"comment":"The headline 7.4 IoU gain compares the half-data DINOv2+Metric3Dv2 Giant model (40.4) with the full-data EfficientNet baseline (33.0), so architecture, pretraining, and training-data volume all vary simultaneously. This does not support the statement that foundation models reduce required training data while surpassing baseline performance. Please report the same-architecture full-data result alongside the half-data result, and provide a matched comparison at fixed iteration or wall-clock budgets, so that the effect of data volume is isolated from the effect of model choice.","section":"§Results: Comparison with Original Model, Table 4"},{"comment":"All reported numbers appear to be single-run peak IoU values selected across training iterations, with no error bars, no multiple seeds, and no explicit checkpoint-selection rule. Given the typical run-to-run variation in BEV segmentation training, differences such as the +1.8 IoU attributed to DINOv2 size or the +3.0 IoU attributed to PseudoLiDAR could be within noise. Please train each key configuration with at least three seeds and report mean and standard deviation, and specify whether the reported IoU is the best validation score during training or a fixed final checkpoint.","section":"§Results: Largeness of Foundation Models, Tables 2–6"},{"comment":"The paper interprets the gains in Tables 4 and 5 as evidence of zero-shot transfer from a frozen generalist depth model, but it never verifies that Metric3Dv2's pretraining corpus excludes nuScenes. Metric3Dv2 is trained on about 8 million images from many datasets, and driving datasets with metric depth are common in such pretraining. If nuScenes appears in that corpus, the improvements are partly memorization of the evaluation domain rather than transfer, and the comparison with an ImageNet-only EfficientNet baseline is asymmetric. Please either provide a list of Metric3Dv2's training datasets with confirmation that nuScenes is excluded, or rerun the key comparisons with a depth model verifiably not trained on nuScenes; an additional cross-dataset evaluation would further strengthen the transfer claim.","section":"§Metric3Dv2 and §Metric Depth Distribution"},{"comment":"The reported iteration counts differ wildly across configurations (e.g., 55k for DINOv2+Metric3Dv2 Giant versus 300k for the baseline). Reporting peak IoU at such different training durations makes it difficult to tell whether the gain comes from the frozen features, the depth signal, or simply from a different effective training schedule. Please report learning curves or at least IoU at matched iteration counts, and state the total compute per configuration, since the Giant models are reported to be roughly 7.5x slower per iteration.","section":"§Results: Comparison with Original Model, Table 4"},{"comment":"The comparison for Simple-BEV is Camera+PseudoLiDAR versus Camera-only, but the PseudoLiDAR cloud is generated from the same camera image using the same intrinsics/extrinsics. The +3.0 IoU therefore quantifies the value of adding this depth-derived representation, yet the paper does not compare against an alternative way of injecting the same depth signal, such as a learnable depth branch or depth-based splatting with matched parameters and compute. The paper cites Simple-BEV's finding that depth splatting loses 3 IoU, but does not implement that variant. Adding such a control would clarify whether the gain is specific to the PseudoLiDAR representation or simply to providing metric depth information.","section":"§Construction of a PseudoLiDAR Point Cloud, Tables 5–6"}],"minor_comments":[{"comment":"Several cross-references are broken: the text says \"described later in Section .\" and \"detailed in Section .\" without numbers, and there are rendering artifacts such as \"F or\" in the abstract.","section":"Throughout"},{"comment":"Reference [7] is a DINOv2 application paper for parasite classification, not the original DINOv2 model paper. The authors should cite Oquab et al., \"DINOv2: Learning Robust Visual Features without Supervision\" (2023), for the model description and pretraining details.","section":"References"},{"comment":"The row \"Giant Giant 41.9 55k\" appears twice; this appears to be a typo, and one copy should be removed.","section":"Table 2"},{"comment":"The PseudoLiDAR depth image resolution is inconsistently reported as (112,200) in Table 5 and Figure 4 but as 112x224 in the text and abstract. Please make the dimensions consistent.","section":"Tables 5–6 and Figure 4"},{"comment":"The names \"Small\", \"Base\", \"Large\", and \"Giant\" for DINOv2 and Metric3Dv2 are not formally defined. Please specify the exact model variants (e.g., ViT-S, ViT-B, ViT-L, ViT-g) and patch sizes so the experiments can be reproduced.","section":"§Results: Largeness of Foundation Models"},{"comment":"There is a typo in \"altered LLS architecture\" where \"LLS\" should be \"LSS\", and the caption of Table 4 says \"LSS\" while the text uses both spellings.","section":"§Results: Comparison with Original Model"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an early-stage empirical report with a useful idea but insufficient experimental rigor for its strong claims. The main risks are the unverified pretraining overlap and the absence of repeated runs; both are fixable within the scope of a revision. I see no indication of bad faith, but the current text overstates the conclusiveness of the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a clean empirical test of dropping frozen DINOv2 and Metric3Dv2 into two standard BEV segmentation pipelines. The specific integrations are new: patching Metric3Dv2 depth into LSS's 41-bin depth distribution, and fusing a Metric3Dv2-derived PseudoLiDAR cloud into Simple-BEV. The ablations over model sizes are useful, and the authors are honest about limitations—small foundation models don't beat the original baseline, the Giant models are roughly 7x slower to train, and depth maps are precomputed.\n\nThe soft spots are substantial. The headline 7.4 IoU gain compares a half-data, foundation-model run against a full-data EfficientNet baseline, so you are changing two variables at once. Every configuration is a single run, with peak IoU selected over training; the reported gaps—like the +3.0 IoU from PseudoLiDAR—are well within the range of seed-to-seed noise for these models. The paper gives no error bars, no seeds, no evaluation protocol details.\n\nThe deeper issue is attribution. The paper treats Metric3Dv2 as a zero-shot generalist, but Metric3Dv2 was trained on roughly 8M images from many driving datasets, and the authors never verify that nuScenes is excluded. If nuScenes is in that corpus, then the gains partly reflect the depth model having already seen the exact sensor geometry and scene distribution of the evaluation set—not transfer from a frozen foundation model. That would undermine the central claim even if every IoU number is reproducible. The same asymmetry applies to DINOv2, though to a lesser degree since its pretraining is more general.\n\nThe authors do show that most of the gain comes from Metric3Dv2: EffNet+Metric3Dv2 Giant reaches 40.5 IoU versus 41.9 with DINOv2, so DINOv2's contribution is small. Reference [10] already explored DINOv2 in BEV, so the novelty is mostly the specific combination and the PseudoLiDAR construction.\n\nThis paper is a legitimate data point for people working on BEV perception with foundation models, but the claims outrun the evidence. A serious referee would ask for multiple seeds, error bars, a clean comparison where only one variable changes, and a check of Metric3Dv2's pretraining data. I would send it to review, but with a clear request for revision.","headline":"Empirical swap of frozen DINOv2/Metric3Dv2 into BEV pipelines shows genuine gains, but single-seed runs and an unverified zero-shot claim for Metric3Dv2 make the headline numbers fragile.","tokens_in":8857,"tokens_out":2412,"would_cite":false,"duration_ms":22327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen DINOv2 and Metric3Dv2 lift BEV vehicle segmentation from 33.0 to 41.9 IoU.","keywords":["birds eye view perception","vehicle segmentation","foundation models","DINOv2","Metric3Dv2","PseudoLiDAR","nuScenes","frozen encoders"],"falsifier":"Train the same top configurations (DINOv2 Giant plus Metric3Dv2 Giant LSS, and Camera plus PseudoLiDAR Simple-BEV) at least five times with different random seeds and report the mean and standard deviation of peak IoU; if the standard deviation exceeds the reported 3 to 6 IoU gaps, or if the baseline catches up within noise, the central claim fails.","tokens_in":1782,"feed_emoji":"🚗","tokens_out":4257,"duration_ms":76129,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf, frozen foundation models can be dropped into existing bird's-eye-view (BEV) perception pipelines to reduce training data and beat the original models. By replacing Lift-Splat-Shoot's EfficientNet encoder with a frozen DINOv2 feature extractor and its learned depth module with Metric3Dv2 metric depth, the authors report vehicle segmentation IoU rising from 33.0 to 41.9 on nuScenes while using half the training data and far fewer iterations. In Simple-BEV, they convert Metric3Dv2 depth images into a PseudoLiDAR point cloud and show it beats the camera-only model by roughly 3 IoU at full image resolution and 6.4 IoU at half resolution. The paper's point is that careful integration of frozen foundation models can modernize old architectures cheaply, with the caveat that the size of the depth model matters more than the size of the feature extractor.","feed_headline":"Frozen foundation models lift BEV segmentation from 33.0 to 41.9 IoU","feed_subtitle":"Replacing learned features and depth with frozen DINOv2 and Metric3Dv2 beats baselines on half the training data.","key_machinery":"The load-bearing mechanism is the frozen foundation-model pair inserted into Lift-Splat-Shoot: DINOv2 supplies 14x14-patch image features with fine granularity, and Metric3Dv2 supplies per-pixel metric depth in meters. To connect Metric3Dv2 to LSS's 41-bin depth distribution, the paper pools continuous depth values into non-overlapping 16x16 patches and assigns each pixel's depth to the corresponding meter bin, producing a tensor of shape 41 by H/16 by W/16; a same-convolutional layer with batch normalization and ReLU stabilizes it. For Simple-BEV, the mechanism is PseudoLiDAR: Metric3Dv2 depth images are unprojected into 3D point clouds using camera intrinsics and extrinsics, then fed through Simple-BEV's standard point-cloud occupancy pathway while retaining bilinear-sampling splatting for image features.","core_discovery":"The core discovery is that replacing the learned components of an established BEV pipeline with frozen foundation models is not only compatible but beneficial. In Lift-Splat-Shoot, substituting EfficientNet with DINOv2 image features and the learned depth distribution with Metric3Dv2's metric depth raised peak vehicle segmentation IoU from 33.0 to 41.9, and the same configuration trained on half the nuScenes training split reached 40.4 IoU, outperforming the original model trained on the full dataset. Ablations show the depth model's scale drives most of the gain: swapping Metric3Dv2 from Small to Giant adds roughly 5.8 IoU, while DINOv2 scale adds about 1.8 IoU. In Simple-BEV, a PseudoLiDAR point cloud generated from Metric3Dv2 depth and projected with camera intrinsics and extrinsics yields 50.7 IoU versus 47.4 for camera-only at 448x800 input, and a larger gain at reduced input resolution, showing the depth signal substitutes for what the camera model failed to infer.","pith_inferences":["We infer that the depth-pooling conversion is generic: any per-pixel monocular depth source could replace Metric3Dv2 in LSS, making this a reusable test harness for future metric-depth models.","An untested corollary is that PseudoLiDAR will help most in low-resolution or degraded-camera settings, since the paper's largest relative gain appears when the camera input resolution is halved.","We note that the paper's single-run protocol leaves the exact magnitudes uncertain; a natural extension is multi-seed training of the top configurations to establish whether the 3 to 6 IoU gaps exceed run-to-run variance.","A further inference is that combining PseudoLiDAR with radar rather than replacing either modality could yield complementary depth cues, though the paper does not test this combination."],"forward_implications":["Frozen DINOv2 plus Metric3Dv2 can replace EfficientNet and learned depth in Lift-Splat-Shoot to gain about 8.9 IoU over the full-data baseline, and still beat that baseline by about 7.4 IoU when trained on half the data.","Metric3Dv2 scale matters more than DINOv2 scale for LSS performance, with the Giant depth model adding roughly 5.8 IoU over the Small one while DINOv2 scale adds about 1.8 IoU.","PseudoLiDAR from Metric3Dv2 depth is a viable cheap substitute for actual LiDAR in Simple-BEV, improving camera-only by about 3 IoU at full resolution and 6.4 IoU at half resolution.","Using frozen foundation models dramatically cuts training iterations: the modified LSS peaks at 55k iterations versus 300k+ for the original and outperforms the original after about 5k iterations.","The approach suggests camera-only BEV perception can reduce dependence on LiDAR hardware, lowering sensor cost while preserving much of the multimodal performance gain."],"supporting_citations":[{"why":"Supplies the baseline Lift-Splat-Shoot architecture whose encoder and depth module are replaced with frozen foundation models.","marker":"[8]"},{"why":"Supplies the Simple-BEV architecture and baselines, including the bilinear-sampling splatting method and the reported LiDAR and radar performance numbers.","marker":"[1]"},{"why":"Provides the frozen DINOv2 feature extractor that replaces EfficientNet in the modified LSS pipeline.","marker":"[7]"},{"why":"Provides the Metric3Dv2 metric depth model used both for depth distributions in LSS and for constructing the PseudoLiDAR point cloud in Simple-BEV.","marker":"[6]"},{"why":"Provides the nuScenes dataset used for training and validation of all compared configurations.","marker":"[2]"}],"fun_headline_variants":["Frozen DINOv2 and Metric3Dv2 push BEV IoU to 41.9","Metric3Dv2 depth as PseudoLiDAR lifts Simple-BEV to 50.7 IoU","Frozen foundation models boost BEV segmentation with half the data","PseudoLiDAR from Metric3Dv2 depth beats camera-only in Simple-BEV"],"cache_read_input_tokens":10880,"weakest_assumption_plain":"The load-bearing premise is that the reported IoU gaps are real and not noise: every configuration is trained once and scored at its peak iteration, so if run-to-run variance is comparable to gaps like the +3.0 or +6.4 PseudoLiDAR improvements, the conclusions could reverse under a different seed.","fun_headline_variants_meta":{"raw":{"variants":["Frozen DINOv2 and Metric3Dv2 push BEV IoU to 41.9","Metric3Dv2 depth as PseudoLiDAR lifts Simple-BEV to 50.7 IoU","Frozen foundation models boost BEV segmentation with half the data","PseudoLiDAR from Metric3Dv2 depth beats camera-only in Simple-BEV"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1861,"prompt_tokens":993,"completion_tokens":868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":771}},"tokens_in":609,"tokens_out":868,"duration_ms":6952,"temperature":1.0,"reasoning_tokens":771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:28:45.024765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same top configurations (DINOv2 Giant plus Metric3Dv2 Giant LSS, and Camera plus PseudoLiDAR Simple-BEV) at least five times with different random seeds and report the mean and standard deviation of peak IoU; if the standard deviation exceeds the reported 3 to 6 IoU gaps, or if the baseline catches up within noise, the central claim fails.","supporting_citations":[{"cited_title":"and Fidler, S","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline Lift-Splat-Shoot architecture whose encoder and depth module are replaced with frozen foundation models."},{"cited_title":"W., Fang, Z., Li, J., Ambrus, R., and Fragki- adaki, K","cited_arxiv_id":null,"evidence_quote":"Supplies the Simple-BEV architecture and baselines, including the bilinear-sampling splatting method and the reported LiDAR and radar performance numbers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the frozen DINOv2 feature extractor that replaces EfficientNet in the modified LSS pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Metric3Dv2 metric depth model used both for depth distributions in LSS and for constructing the PseudoLiDAR point cloud in Simple-BEV."},{"cited_title":"H., V ora, S., Liong, V","cited_arxiv_id":null,"evidence_quote":"Provides the nuScenes dataset used for training and validation of all compared configurations."}],"review_version":1}