{"id":"f8e05072-214b-4b9a-a8ab-f18ecb38f6bc","arxiv_id":"2412.12460","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"PromptDet is a single-stage framework whose LiDAR prompter improves both fusion and camera-only 3D detection on nuScenes with under 2% additional parameters.","lead":"PromptDet adds a lightweight LiDAR-powered prompter to camera-based 3D detectors, fusing point clouds and images at multiple scales and distilling the fused knowledge back into the camera branch. It improves both LiDAR-camera detection (up to 22.8 mAP) and camera-only detection (up to 2.4 mAP) on nuScenes with under 2% extra parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Camera-only gains are conditional on LiDAR supervision during training; the abstract's 'Without LiDAR points' wording can mislead, so the camera-only claim should be scoped to inference-only.","rationale":"The reader identified the dependency on paired LiDAR-camera training data as the weakest assumption, and I agree. This is the most load-bearing concern because the camera-only improvement is a headline result, and the paper's phrasing could overstate the method's applicability. The concern is not a fatal flaw: the paper does specify the inference-time-only nature of the camera-only mode, and the ablations partially support the design. However, the missing control experiment (training without LiDAR correspondence) and the unmeasured camera-only latency leave open the possibility that the advertised camera-only gains are partly due to factors other than LiDAR-specific knowledge or that the inference-time claim is inaccurate. These gaps justify the reader's conditional verdict without requiring a rejection. I recommend keeping the verdict unchanged: conditional acceptance, with the request that the authors clarify the training-time LiDAR requirement in the abstract and report camera-only latency plus the LiDAR-shuffled control experiment.","tokens_in":13754,"tokens_out":11674,"duration_ms":105689,"concrete_test":"Train BEVDet+PromptDet on a 'LiDAR-shuffled' nuScenes variant where each training sample is paired with a random point cloud from a different scene, breaking the geometric correspondence between LiDAR and the surrounding images. If the camera-only mAP/NDS drops back to the baseline level, this confirms that the camera-only gains depend on the paired LiDAR-derived geometric signal rather than on the joint multi-task training objective alone. Second, directly measure the camera-only inference latency (with LiDAR branch removed) and compare it with the baseline latency to verify the 'almost no impact' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim includes a camera-only improvement of up to 2.4 mAP and 4.0 NDS with 'almost no impact' on inference time. This improvement is only obtained in an inference-time setting where a LiDAR branch was present during training. The method has no mechanism for improving a camera-only detector when LiDAR is never available at training time: CMKI requires a fusion branch to exist, and AHA requires LiDAR features. The abstract's phrase 'Without LiDAR points, PromptDet still achieves an improvement' could be read as a fully camera-only pipeline, but the method is a training-time multi-modal approach with an inference-time camera-only mode. This is a boundary condition rather than an internal inconsistency, but it is load-bearing because any deployment without LiDAR in the training data would not obtain the advertised gains. The paper states this dependency in the methodology ('There are two situations during PromptDet inference') but does not flag it prominently in the abstract or conclusion, risking a misleading interpretation. Additionally, the 'almost no impact' on camera-only inference time is not directly measured in Table 5; only the multi-modal (L&C) latency is reported, so the inference-time claim is unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PromptDet, a plug-and-play LiDAR-assisted prompter for BEVDet-family camera-based 3D detectors. The method uses an Adaptive Hierarchical Aggregation (AHA) module to fuse LiDAR voxel features and camera pseudo-voxel features at three voxel scales, and a Cross-Modal Knowledge Injection (CMKI) module that distills knowledge from the detached fusion branch into the camera branch using UniDistill-style losses. The framework is trained in a single stage with a hybrid supervision strategy in which both branches share the BEV encoder and detection head. Experiments on nuScenes show substantial multi-modal gains (up to 22.8 mAP points and 21.1 NDS points with fewer than 2% extra parameters) and smaller camera-only inference gains (up to 2.4 mAP and 4.0 NDS), with the claim that the camera-only gains come with almost no impact on inference time.","tokens_in":13992,"tokens_out":8251,"duration_ms":66887,"significance":"The core contribution is a parameter-efficient fusion module that can be added to existing BEVDet-series detectors, with empirical validation across three baselines, component ablations, hyperparameter sensitivity, and efficiency comparisons. The released code and the reproducibility of the experiments are strengths. The paper's framing as 'prompt learning' is mostly analogical, but the technical content—online cross-modal distillation with gradient detachment—is sound and useful. The main value is practical: a user can obtain multi-modal fusion performance and an improved camera-only mode from a single-stage training run.","major_comments":[{"comment":"The camera-only improvement reported in the abstract ('Without LiDAR points, PromptDet still achieves an improvement...') is only available when LiDAR data were present during training, because CMKI requires a fusion branch and AHA requires LiDAR features. The methodology section states this dependency, but the abstract and conclusion do not; please scope the claim explicitly, for example by saying 'when LiDAR is used during training but is unavailable at inference, PromptDet-C remains improved.'","section":"Abstract; §Training and Inference"},{"comment":"The claim that PromptDet-C has 'almost no impact on camera detection inference time' is not supported by any measurement. Table 5 reports latency only for the L&C mode of +LaP, not for the camera-only branch. Please measure and report the inference latency of the camera-only baseline with and without the imitation module, or qualify the claim as an estimate based on the two added convolution layers.","section":"§Model Efficiency; Table 5"},{"comment":"The increases of 22.8% mAP and 21.1% NDS are absolute percentage-point differences (e.g., 50.5 vs. 27.7 mAP in Table 2), not relative improvements. Please use 'percentage points' throughout the abstract, introduction, and main text to avoid misreading.","section":"Abstract; Table 2"},{"comment":"The paper motivates PromptDet by the training cost of KD-based methods, but no experiment measures training time, memory, or FLOPs against a KD baseline such as UniDistill or SimDistill. Since the 'single-stage, lightweight' claim is central to the positioning, please add such a comparison or temper the qualitative claims about training cost.","section":"Introduction; §Model Efficiency"}],"minor_comments":[{"comment":"The cross-references in the first paragraph of §Methodology are empty ('In Section , we introduce'). Please insert the correct section numbers.","section":"Methodology"},{"comment":"The 'LiDAR modality switch' semantics are confusing: in §Training and Inference, the switch is 'turned off' for multi-modal detection and 'turned on' for camera-only detection, which inverts the usual meaning of a switch. Please rename it (e.g., 'LiDAR bypass switch') and describe its state unambiguously.","section":"Figure 2; §Training and Inference"},{"comment":"There is a typo, 'PrompDet', in the caption of Figure 2.","section":"Figure 2 caption"},{"comment":"The 'dynamic point cloud voxelization' is attributed to (Zhou et al. 2020), but that reference appears to describe an end-to-end multi-view fusion method rather than dynamic voxelization; please clarify the reference or cite the correct source.","section":"Experimental Setup"}],"recommendation":"minor_revision","confidential_remarks":"The paper's central technical contribution is sound and the experiments are extensive. The main concerns are about scoping of the camera-only claim and a missing latency measurement. The paper fits the journal's scope; no citation concerns. I recommend minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PromptDet is a solid engineering contribution. It adds a lightweight LiDAR fusion branch to BEVDet-family camera detectors, and the gains are large on nuScenes: up to 22.8 points mAP and 21.1 points NDS over the camera-only baseline with under 2% extra parameters. The camera-only branch also improves after training, by up to 2.4 mAP and 4.0 NDS, which is the interesting part. The core idea—online self-distillation from the fusion branch to the camera branch, sharing the same encoder and head, with no pretrained teacher—is genuinely new relative to the cited KD methods, which all need a trained teacher. The experiments span three base detectors and the ablations isolate each component. I found no fatal flaw in the method or the numbers.\n\nThe main soft spot is the scope of the camera-only claim. The abstract says 'Without LiDAR points, PromptDet still achieves an improvement,' and that is only true for inference. Training still requires paired LiDAR and camera data: CMKI needs the fusion branch, and AHA needs LiDAR features. If you train on images alone, you get none of the camera-only gain. The paper does state this in the methodology, but the abstract and conclusion gloss over it. That is a boundary condition, not an internal inconsistency, but it is load-bearing and should be flagged prominently.\n\nTwo minor issues. First, the 'almost no impact on camera detection inference time' is not supported by a table; Table 5 only reports latency for the multi-modal variants. The camera-only branch adds two convolution layers, so the claim is plausible, but it should be measured and reported. Second, the 22.8% and 21.1% improvements are absolute percentage-point gains, and the paper should say so to avoid ambiguity. Also, the 'prompt' framing is a bit of a stretch because the base camera model is not frozen; the method is really a lightweight fusion plus self-distillation design. These are cosmetic compared to the main caveat.\n\nI would send this to peer review. The engineering is sound, the result is useful for practical systems, and the revisions needed are straightforward: scope the camera-only claim to inference after multi-modal training, add the latency measurement, and clarify the percentage-point language. The paper deserves referee time.","headline":"Solid engineering result: a lightweight LiDAR fusion branch that also improves camera-only inference, but the camera-only gains are conditional on LiDAR at training time and the paper should say so more clearly.","tokens_in":14500,"tokens_out":3766,"would_cite":true,"duration_ms":31695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PromptDet treats LiDAR as a prompt, adding fewer than 2% parameters to camera-only BEV detectors; it reports gains up to 22.8 mAP and 21.1 NDS in multi-modal mode and up to 2.4 mAP and 4.0 NDS in camera-only mode.","keywords":["3D object detection","multi-modal fusion","prompt learning","bird's-eye view","LiDAR-camera fusion","knowledge distillation","nuScenes","autonomous driving"],"falsifier":"Train PromptDet on a camera-only dataset (no LiDAR at training time) and check whether the camera branch still beats the baseline; if it does, the gains are not from cross-modal knowledge, and if it does not, the paper's camera-only claim is confirmed to require LiDAR supervision. Alternatively, replace the fusion features used in CMKI with a fixed random noise feature of the same shape and retrain; any remaining camera-only gain would imply the loss is a regularizer rather than a vehicle for LiDAR geometry.","tokens_in":13538,"feed_emoji":"🚗","tokens_out":5886,"duration_ms":49777,"temperature":0.7,"pith_summary":"The paper tries to establish that LiDAR does not need a full fusion network or a pre-trained teacher to help a camera-based 3D detector. The missing depth geometry can be injected through a small 'prompter' made of two modules, AHA and CMKI, leaving the base detector nearly unchanged. If this is right, a single one-stage training run yields two usable detectors: a strong LiDAR-camera fusion detector and an improved camera-only detector that still beats the original camera baseline. The reported gains on nuScenes are up to 22.8% mAP and 21.1% NDS for fusion, and up to 2.4% mAP and 4.0% NDS for camera-only inference, with parameter growth below 2% and nearly unchanged camera inference time.","feed_headline":"LiDAR as a prompt lifts 3D detection by 22.8 mAP","feed_subtitle":"A small prompter on BEVDet-family cameras also improves camera-only results by 2.4 mAP at no inference cost.","key_machinery":"Adaptive Hierarchical Aggregation (AHA) and Cross-Modal Knowledge Injection (CMKI) together form the LiDAR-assisted prompter. AHA computes softmax attention weights over pairs of LiDAR voxel features and camera pseudo-voxel features at three voxel scales, combines them, and flattens the result into a fusion BEV feature. CMKI first maps the camera BEV feature through a 3D-then-2D convolution imitation module so it can mimic the fusion feature, then applies three distillation losses (feature, relation, response) from detached fusion features to camera features; detaching ensures fusion training is not dragged down by the camera branch. A LiDAR switch at inference selects which features enter the shared BEV encoder and detection head.","core_discovery":"The central claim is that LiDAR geometry can be treated as a prompt rather than as a second stream: the LiDAR-assisted prompter fuses point-cloud voxel features with camera pseudo-voxel features at three scales using learned attention weights (AHA), then uses the fused features to teach the camera branch online through feature, relation, and response distillation (CMKI). By supervising both branches with the same ground truth and detaching the fusion branch from the distillation gradients, the same network learns to answer with and without LiDAR. On the nuScenes benchmark, the authors report that this recipe improves BEVDet-family baselines by up to 22.8 mAP and 21.1 NDS in multi-modal mode with fewer than 2% extra parameters, and by up to 2.4 mAP and 4.0 NDS in camera-only mode with negligible inference overhead.","pith_inferences":["The method could likely be reused with other auxiliary depth signals (e.g., radar or pseudo-LiDAR) as the 'prompt' source, since the prompter only needs voxel features aligned to the camera BEV space.","A testable extension is whether the camera-only gain is a distillation of depth/location information or just a regularizing effect of multi-task training; comparing against a camera-only model trained with the same losses but a scrambled or random fusion branch would isolate the LiDAR-specific contribution.","Because the gains are measured on nuScenes, the method's practical value depends on how well the learned prompting transfers to new cities, camera rigs, or weather; a domain-shift benchmark would reveal how much of the improvement is dataset-specific.","The paper's parameter counts exclude LiDAR preprocessing, so the reported '+2%' is about network parameters only; an editorial reader should separate model size from total system cost when comparing with dedicated fusion detectors."],"forward_implications":["Multi-modal 3D detection can be bolted onto an existing camera detector in one training stage, without pretraining a separate teacher and without a second heavy backbone.","The same checkpoint covers both fusion inference and camera-only inference, so LiDAR failure at deployment need not disable detection or require a fallback model.","The camera-only branch's improvement transfers learned geometry into features, so under the reported settings it costs almost no extra inference time.","Because the prompter is plug-and-play, it can be expected to extend to other BEVDet-series detectors and to related multi-camera perception heads such as occupancy prediction and BEV segmentation (the paper suggests this)."],"supporting_citations":[{"why":"Provides BEVDet, the camera-only baseline architecture that PromptDet extends.","marker":"Huang et al. 2021"},{"why":"Provides BEVDepth, a base model with explicit depth supervision used to validate generalization and as a main comparison model.","marker":"Li et al. 2023a"},{"why":"Gives BEVFusion, the fusion baseline and the efficiency comparison that PromptDet outperforms.","marker":"Liang et al. 2022"},{"why":"Defines UniDistill, whose feature, relation, and response distillation losses CMKI adapts for online knowledge injection.","marker":"Zhou et al. 2023"},{"why":"Supplies the nuScenes dataset and the mAP/NDS metrics used in all reported experiments.","marker":"Caesar et al. 2020"},{"why":"Provides the LSS lifting operation that creates the camera pseudo-voxel features in the BEVDet-series baselines.","marker":"Philion and Fidler 2020"}],"fun_headline_variants":["LiDAR as a prompt lifts 3D detection up to 22.8 mAP","Lightweight LiDAR prompts improve 3D detection","Turn LiDAR into prompts for 3D detection","LiDAR prompts: +22.8 mAP multi-modal, +2.4 mAP camera-only","No heavy fusion: LiDAR as a prompt for 3D detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The camera-only improvements only appear when paired LiDAR and camera data are available during training; if LiDAR is absent from the training set, the framework gives the camera branch no extra knowledge to learn.","fun_headline_variants_meta":{"raw":{"variants":["LiDAR as a prompt lifts 3D detection up to 22.8 mAP","Lightweight LiDAR prompts improve 3D detection","Turn LiDAR into prompts for 3D detection","LiDAR prompts: +22.8 mAP multi-modal, +2.4 mAP camera-only","No heavy fusion: LiDAR as a prompt for 3D detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001162,"raw_usage":{"total_tokens":4864,"prompt_tokens":1052,"completion_tokens":3812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":3713}},"tokens_in":668,"tokens_out":3812,"duration_ms":25285,"temperature":1.0,"reasoning_tokens":3713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:03:16.391894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PromptDet on a camera-only dataset (no LiDAR at training time) and check whether the camera branch still beats the baseline; if it does, the gains are not from cross-modal knowledge, and if it does not, the paper's camera-only claim is confirmed to require LiDAR supervision. Alternatively, replace the fusion features used in CMKI with a fixed random noise feature of the same shape and retrain; any remaining camera-only gain would imply the loss is a regularizer rather than a vehicle for LiDAR geometry.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines UniDistill, whose feature, relation, and response distillation losses CMKI adapts for online knowledge injection."}],"review_version":1}