{"id":"1d2f3eb9-c228-4817-ad5d-f14af9a27831","arxiv_id":"2504.12709","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Pre-training on combined unlabeled NuScenes, Lyft, and ONCE data with BEV contrastive learning, image MAE, and dataset prompts improves downstream 3D perception tasks.","lead":"Self-supervised pre-training on combined unlabeled camera and LiDAR datasets improves 3D detection, tracking, segmentation, and occupancy prediction in autonomous driving. The paper combines BEV contrastive learning, image masked autoencoding, and dataset prompts to make mixed-dataset pre-training work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling evidence in Table 6 is confounded: the '250K' run presents roughly 403K samples per epoch, and most metric increments are within noise, so the central 'steady scaling' claim is not established.","rationale":"I read the paper in good faith. The core pre-training result in Table 1—detection mAP 51.5→55.5 and NDS 53.5→58.8 over scratch, with ablations showing both losses contribute—is directionally plausible, and the prompt-vs-naive-mixing comparison in Tables 4 and 5 is internally consistent. The BEV correspondence assumption identified by the reader is a genuine concern, but it is not the single most load-bearing issue: within each sample, LSS uses that dataset's own extrinsics, so the same-position positive pairs are at least geometrically defined, and the empirical gains suggest the alignment is not catastrophically wrong. The scaling claim is different. Table 6 is the only evidence for the paper's central scalability narrative, and it has two unaddressed problems: the reported metrics are mostly flat or within error bars, and the '250K' run actually feeds the model about 403K samples per epoch because NuScenes and Lyft are repeated 4×. That confound means even the small numerical improvement from 100K to 250K could come from additional optimization steps rather than from scaling unique data. Because the paper explicitly presents 'the potential to continually benefit 3D perception models' as a contribution, this confound directly threatens a headline claim and should be resolved before the scaling conclusion is accepted. The reader's verdict of CONDITIONAL remains appropriate—the preprint's main pre-training idea is promising, but the scaling evidence must be cleaned up and, ideally, code released for independent verification. My concern does not move the verdict; it sharpens the conditions under which the claim should be accepted.","tokens_in":18677,"tokens_out":7269,"duration_ms":73531,"concrete_test":"Re-run the Table 6 scaling experiment with the total number of optimizer steps and the per-epoch sample count held constant across the 40K, 100K, and 250K conditions—for example, subsample ONCE so that each condition presents exactly 100,810 samples per epoch, and train all three for the same number of epochs. Then report the mAP, NDS, AMOTA, and BEV-segmentation mIoU with the same repeated-seed protocol as Table 1. If the 250K condition no longer improves over 100K (or improves only within standard deviations), the paper's scaling claim is unsupported. Additionally, report the exact epoch count used for each row of Table 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline scalability claim—'steady performance increase as the training data volume scales up' (abstract, RQ3)—rests entirely on Table 6, and that evidence is not clean. First, the reported curves are not monotonic: mAP is 53.6±0.7 at 40K and 53.6±0.4 at 100K; NDS moves 55.8±1.8 → 56.0±0.5 → 56.1±0.3; AMOTP worsens at 100K (78.2 vs 77.1 at 40K); occupancy mIoU is flat at 16.1 for 100K and 250K. Most increments are within one standard deviation, so 'steady improvement' is not demonstrated by the published numbers. Second, the data-volume axis is confounded. Appendix Table 8 reports unique-frame totals of roughly 39.9K, 100.8K, and 250.8K for the three runs, but for the 250K run NuScenes (28,130 frames) and Lyft (22,680 frames) are repeated 4×, so each epoch presents 112,520 + 90,720 + 200,000 = 403,240 samples—about 4× the 100K run's per-epoch count (100,810) and about 10× the 40K run's count. If the number of epochs is held fixed, the 100K→250K improvement can be attributed to 4× more optimizer steps and repeated exposure to the same NuScenes/Lyft frames, not to scaling unique data volume. The paper does not state the epoch count or total step budget for Table 6, so the scaling conclusion is not well defined. This is load-bearing because the paper's 'potential to continually benefit / foundation-model' narrative depends on a clean scaling curve that the current experimental design cannot provide.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised pre-training framework for multimodal (camera + LiDAR) Bird's Eye View (BEV) perception in autonomous driving. The method combines a masked autoencoder (MAE) loss on images with a contrastive loss that treats corresponding grids in LSS-projected image BEV and LiDAR BEV maps as positive pairs, and it uses dataset-specific prompt adapters to mitigate domain bias when pre-training on a mixture of NuScenes, Lyft, and ONCE data. The authors pre-train Swin-Transformer and DSVT backbones from scratch, then fine-tune on 20% of NuScenes or 20% of Waymo and evaluate on 3D object detection, 3D object tracking, BEV segmentation, and occupancy prediction. They report improvements over Scratch and a modified BEVDistill baseline, show that the prompt adapter helps over naive dataset mixing, and claim steady performance improvement as pre-training data volume scales up to 250K frames.","tokens_in":19115,"tokens_out":6829,"duration_ms":69783,"significance":"If the main results hold, the paper is a useful empirical contribution: it demonstrates that heterogeneous unlabeled multi-dataset pre-training can improve multimodal BEV perception across several downstream tasks, and it provides an explicit mechanism (prompt adapters) for reducing dataset bias during self-supervised pre-training. The evaluation across four tasks, the held-out Waymo experiment, the ablation of both losses, and the correct-versus-wrong prompt comparison are concrete strengths. The scalability claim, however, is not supported by the present experimental design, and the held-out improvement is borderline relative to noise. The paper is worth publishing after the scaling experiment and the alignment assumption are addressed.","major_comments":[{"comment":"The scalability claim is confounded. Appendix Table 8 shows that the 250K-frame run actually presents 28,130*4 + 22,680*4 + 200,000 = 403,240 samples per epoch, versus 100,810 for the 100K run and approximately 39,900 for the 40K run. Since Appendix A states that Tables 4-6 keep the same settings except for the learning rate, the 250K run is trained with about four times as many optimizer steps as the 100K run and repeats the NuScenes and Lyft frames four times. The reported curves are also non-monotonic and mostly within one standard deviation: mAP is 53.6±0.7 -> 53.6±0.4 -> 54.1±0.6, AMOTP worsens from 77.1 at 40K to 78.2 at 100K before returning to 77.1, and occupancy mIoU is flat at 16.1 for both 100K and 250K. The abstract and conclusion claim of \"steady performance increase\" is therefore not established. Please rerun with a matched total compute/step budget, or at minimum report the number of epochs and total steps for each run and separate unique-data scaling from repeated-sample/epoch scaling.","section":"Section 4.4, Table 6, Appendix Table 8"},{"comment":"The contrastive loss treats BEV grids at corresponding spatial positions in LSS-projected image BEV and LiDAR BEV as positive pairs. This assumes that the camera-derived BEV is geometrically aligned with the LiDAR BEV across datasets with different camera rigs, mounting angles, and fields of view, and that occlusion or missing sensor coverage does not create systematic mismatches. The paper does not quantify the alignment error or ablate the correspondence definition. If the correspondence is noisy, the contrastive loss can encode alignment artifacts rather than semantic knowledge. Please provide quantitative alignment diagnostics (e.g., mean distance between LSS-projected image features and LiDAR occupancy per grid) or an ablation with an alternative correspondence or negative-sampling rule.","section":"Section 3.1, Eqs. (2)-(3)"},{"comment":"The held-out Waymo evidence is weaker than the text suggests. The improvement over Scratch is only 0.7 L1 mAP (66.6±0.8 vs. 65.9±1.1), which is within the reported error bars, and the BEVDistill baseline is below Scratch (64.8±1.1). The statement that the method \"effectively enhances model performance in out-of-distribution scenarios\" overstates this evidence. Please report significance tests or more seeds, or soften the claim accordingly.","section":"Section 4.2, Table 3"}],"minor_comments":[{"comment":"The text introduces a \"Random\" setting for prompt handling that does not appear in Table 5; please clarify or remove it.","section":"Appendix C.1"},{"comment":"Several key hyperparameters are missing: the NCE temperature tau, the MAE masking ratio, the number of sampled grids K, and the relative weights of L_MAE and L_CL in Eq. (1). These should be reported for reproducibility.","section":"Appendix A and Eq. (3)"},{"comment":"The table header is garbled (three identical \"NuScenes\" columns); it should label NuScenes, Lyft, and ONCE separately.","section":"Appendix Table 8"},{"comment":"There are several typos and citation issues: \"rescent\" should be \"recent\", \"have have\" appears in the contrastive-learning discussion, and reference [68] is cited as an MAE method although the listed paper is M2BEV, which is not a masked autoencoder method.","section":"Related Works"},{"comment":"The text contains typos such as \"bechmark\", \"portions\", \"detailes\", and \"fro occupancy\"; these should be corrected.","section":"Section 4.1"},{"comment":"The only pre-training baseline is a modified version of BEVDistill with the object-level distillation loss removed, and CALICO is not compared because it is not open-sourced. A comparison with other recent self-supervised BEV pre-training methods, if available, would strengthen the claim, and the unmodified BEVDistill result should be reported if feasible.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The core pre-training and prompt-adapter results in Tables 1, 2, 4, and 5 are plausible and internally consistent, but the scalability experiment in Table 6 is the weakest load-bearing point: as designed it cannot distinguish data-volume scaling from step-count scaling. This is fixable within the scope of the paper, so I recommend major revision rather than rejection. The geometric-alignment assumption in Eq. (3) also deserves explicit analysis before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2504.12709. The new bit is combining known pieces—BEV contrastive learning between image and LiDAR, image MAE, and dataset-specific prompt adapters—into one self-supervised pretraining pipeline for multimodal 3D perception on multiple unlabeled datasets. That exact combination is not in the literature, and the empirical work is reasonably broad: four downstream tasks, a held-out Waymo detection experiment, ablations for both losses, and a \"wrong prompt\" check that suggests the prompt adapter is doing something real. The main tables are internally consistent apart from the issue below, and the finding that naive mixing hurts without prompts but helps with prompts is a useful datapoint.\n\nNow the soft spots, in proportion. The biggest is the scaling claim (RQ3/Table 6). The numbers do not show a steady increase: mAP is flat 53.6→53.6→54.1, NDS moves 55.8→56.0→56.1, and several increments are within one standard deviation. Worse, the data-volume axis is confounded. Appendix Table 8 shows the 250K run presents about 403K samples per epoch, roughly 4x the 100K run and 10x the 40K run, because NuScenes and Lyft are repeated 4x while ONCE jumps from 50K to 200K unique frames. If epochs are held constant (the paper doesn't say), the 250K gain could be from 4x more optimizer steps, not unique data scaling. The abstract's \"steady performance increase as the training data volume scales up\" is therefore not established.\n\nThere's also a red flag I didn't see in the reader notes: Table 1's \"Ours\" (pretrained on NuScenes, fine-tuned on 20% NuScenes) gives mAP 55.5, NDS 58.8, but Table 4's \"Single dataset\" row, described the same way, gives mAP 53.3, NDS 54.2. Table 4 uses lr 2e-4 instead of 4e-4, but a learning rate change shouldn't cost 4.6 NDS points if the setup is otherwise identical. Either the two setups differ in some unstated way, or one of the tables is wrong. Needs clarification.\n\nMinor issues: only one baseline (modified BEVDistill, with CALICO not compared), missing hyperparameters (temperature τ, masking ratio, K, loss weights, repeat ratios), and the BEV grid correspondence assumption (position-matching between LSS-projected image BEV and LiDAR BEV) is untested. None of these sink the core pretraining claim, but they limit confidence. No code is available yet despite the abstract saying it will be released.\n\nBottom line: this is a workable recipe paper, but not a scaling-law/foundation-model story. Send it to peer review, but ask for a fix to the scaling experiment (report unique samples and step budget; either show a clean curve or soften the claim), resolution of the Table 1 vs Table 4 inconsistency, stronger baselines, and hyperparameters. With those, it would be a solid contribution.","headline":"The combination of BEV contrastive learning, image MAE, and dataset prompt adapters is a sensible recipe with promising held-in results, but the headline scaling claim is not supported by Table 6, and an unexplained inconsistency between Tables 1 and 4 needs resolution.","tokens_in":19644,"tokens_out":3688,"would_cite":false,"duration_ms":34543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised pre-training on combined unlabeled LiDAR-camera datasets lifts 3D perception across detection, tracking, segmentation, and occupancy, and scales with data volume.","keywords":["self-supervised pre-training","3D object detection","BEV segmentation","object tracking","occupancy prediction","multi-dataset training","domain adaptation","prompt adapter"],"falsifier":"Corrupt the correspondence by shifting the image-derived BEV grid relative to the LiDAR grid by a fixed offset (e.g., 2 meters) during pre-training while keeping everything else unchanged; if downstream gains persist, the learned alignment is not the source of the improvement, and the claimed mechanism is not load-bearing.","tokens_in":18510,"feed_emoji":"🚗","tokens_out":7279,"duration_ms":65919,"temperature":0.7,"pith_summary":"Self-supervised pre-training on unlabeled LiDAR-camera data from multiple driving datasets can substitute for some annotated 3D data. The paper proposes a framework that jointly trains image and point-cloud backbones from scratch using a BEV-grid contrastive loss and an image masked-autoencoder loss, then fine-tunes on a small annotated subset. Adding per-dataset learnable prompts through a prompt adapter mitigates the domain gap between heterogeneous datasets, so that combining NuScenes, Lyft, and ONCE frames improves rather than hurts downstream performance. Across 3D object detection, 3D tracking, BEV segmentation, and occupancy prediction, the approach beats training from scratch and a BEVDistill baseline, and performance rises steadily as pre-training data grows from 40K to 250K frames.","feed_headline":"Self-supervised pre-training lifts 3D perception on four tasks","feed_subtitle":"Unlabeled frames from mixed driving datasets replace annotated 3D labels and scale with data volume.","key_machinery":"The central mechanism is a BEV-grid contrastive loss: image features are projected into a bird's-eye-view grid via lift-splat-shoot (LSS), LiDAR features are voxelized into the same BEV grid, and the corresponding grid cells across modalities are treated as positive pairs under an NCE loss, with non-corresponding cells as negatives. A masked-autoencoder (MAE) loss on the image patches supplements this, preserving semantic detail lost when images are lifted to BEV. To handle dataset bias, each dataset gets a learnable soft prompt that an MLP adapter converts into scale and bias parameters modulating LayerNorm layers (PromptNorm), so the backbone shares common knowledge while dataset-specific variation is absorbed outside it.","core_discovery":"The paper claims that a self-supervised pre-training objective based on cross-modal BEV alignment plus image reconstruction lets a multi-modal 3D perception model learn transferable representations entirely from unlabeled driving data. On 20% of NuScenes fine-tuning data, the pre-trained model outperforms a from-scratch baseline and a BEVDistill-style baseline on object detection (mAP 55.5 vs 51.5 vs 54.1), tracking (AMOTA 61.8 vs 57.1 vs 60.6), BEV segmentation (mIoU 39.3 vs 39.1 vs 39.1), and occupancy prediction (IoU 32.5 vs 32.3 vs 32.3), and it also generalizes out-of-domain to Waymo detection. The paper further claims that naive mixing of multiple datasets degrades performance, while per-dataset prompt adapters restore and improve it, and that scaling the unlabeled pre-training set from 40K to 250K frames yields steady gains across all tasks.","pith_inferences":["The BEV-grid correspondence definition is tested only through its downstream effect; a direct measurement of alignment error (e.g., mean distance between projected camera rays and LiDAR returns per cell) would sharpen the claim and likely reveal where the method breaks.","The same prompt-adapter idea could be applied at fine-tuning time to adapt a single pre-trained model to new cities or sensor configurations without retraining the backbone, which the paper only partially explores.","The scaling trend suggests that with more data the contrastive-and-MAE objective may approach the behavior of a foundation model for driving scenes, but the paper's largest run (250K frames) is still small relative to the 1M+ frames available in ONCE.","An immediate testable extension is to ablate the correspondence by replacing positive pairs with random pairs within a local neighborhood, isolating whether the spatial alignment or the mere cross-modal agreement drives the gain."],"forward_implications":["Fine-tuning on as little as 20% of annotated NuScenes reaches detection mAP 55.5 and NDS 58.8, suggesting that unlabeled pre-training can cut annotation cost.","Pre-training on combined datasets with prompt adapters outperforms single-dataset pre-training, so heterogeneous unlabeled data can be pooled rather than discarded.","Performance rises monotonically from 40K to 250K pre-training frames across all four tasks, indicating the method has room to scale with data volume.","The learned representations transfer out-of-domain (NuScenes pre-training to Waymo fine-tuning) with L1 mAP 66.6, beating both scratch and the BEVDistill baseline.","Even initializing a downstream prompt with the 'wrong' dataset prompt still outperforms training without prompts, implying the backbone itself stores generalizable cross-dataset knowledge."],"supporting_citations":[{"why":"Supplies the primary pre-training and fine-tuning data (NuScenes) and the evaluation benchmark for all four downstream tasks.","marker":"[3]"},{"why":"Contributes a second heterogeneous unlabeled dataset to the multi-dataset pre-training combination.","marker":"[11]"},{"why":"Provides the large-scale unlabeled source (up to 200K sampled frames) used for the scaling experiments.","marker":"[34]"},{"why":"Is the open-source pre-training baseline the method must beat and adapts by removing the object-level distillation loss that depends on labels.","marker":"[9]"},{"why":"Supplies the masked-autoencoder loss that preserves image semantic information in the pre-training objective.","marker":"[17]"},{"why":"Defines the lift-splat-shoot projection used to bring image features into the BEV grid for the contrastive loss.","marker":"[43]"},{"why":"Supplies the prompt-adapter design (dataset-specific prompts injected into LayerNorm) that mitigates the domain gap in multi-dataset training.","marker":"[67]"}],"fun_headline_variants":["Unlabeled data pre-training boosts 3D perception","Self-supervised pre-training on unlabeled driving data scales","Mixed-dataset pre-training lifts 3D perception without labels","Pre-training on unlabeled frames improves 3D perception tasks","Self-supervised pre-training: better 3D perception from unlabeled data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BEV grids from camera projection and LiDAR genuinely correspond cell-by-cell across all datasets; if sensor calibration, occlusion, or differing camera rigs make those correspondences systematically wrong, the contrastive loss learns alignment artifacts rather than semantic knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Unlabeled data pre-training boosts 3D perception","Self-supervised pre-training on unlabeled driving data scales","Mixed-dataset pre-training lifts 3D perception without labels","Pre-training on unlabeled frames improves 3D perception tasks","Self-supervised pre-training: better 3D perception from unlabeled data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1319,"prompt_tokens":911,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":322}},"tokens_in":527,"tokens_out":408,"duration_ms":4097,"temperature":1.0,"reasoning_tokens":322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:24:23.820876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt the correspondence by shifting the image-derived BEV grid relative to the LiDAR grid by a fixed offset (e.g., 2 meters) during pre-training while keeping everything else unchanged; if downstream gains persist, the learned alignment is not the source of the improvement, and the claimed mechanism is not load-bearing.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the masked-autoencoder loss that preserves image semantic information in the pre-training objective."},{"cited_title":"Lift, splat, shoot: En- coding images from arbitrary camera rigs by implicitly un- projecting to 3d","cited_arxiv_id":null,"evidence_quote":"Defines the lift-splat-shoot projection used to bring image features into the BEV grid for the contrastive loss."},{"cited_title":"Towards large-scale 3d representation learning with multi-dataset point prompt training","cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-adapter design (dataset-specific prompts injected into LayerNorm) that mitigates the domain gap in multi-dataset training."}],"review_version":1}