{"id":"f0c0d9cf-1012-4343-8039-cb2cc46ff510","arxiv_id":"2507.20397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"VESPA fuses LiDAR geometry with vision-language model semantics to generate open-vocabulary 3D pseudolabels, achieving 52.95% class-agnostic AP and 46.54% 3-class mAP on nuScenes without human supervision.","lead":"VESPA is a pipeline that automatically labels 3D LiDAR point clouds for self-driving cars by combining camera images with vision-language models, without needing human annotations. If it holds up, it could make 3D data labeling for autonomous driving far cheaper, since the system reports state-of-the-art results on the nuScenes benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA over UNION rests on a reimplemented baseline whose fidelity is unverified; if official UNION scores are higher, the 12.5-point AP gain and the central claim are not supported.","rationale":"Both the reader and I read the paper as a strong empirical pipeline with credible ablations; the central claim is an empirical comparison claim, so its weakest link is the comparison itself. The paper's own Table 1 footnote admits UNION is a reimplementation; the original UNION paper's results are not quoted, and no configuration is given. This matters more than the DBSCAN threshold: Table 4 proves DN improves mAP by 19 points, which supports the module's effect, whereas the magnitude of the gap over UNION is entirely dependent on the reimplementation's fidelity. A threshold value being unreported is a reproducibility defect that can be fixed by a table; a baseline that may be undertuned can invalidate the headline. I am not claiming the reimplementation is wrong; I am claiming the paper provides no check that it is right. The concrete test is to compare against the official released UNION. This does not change the reader's CONDITIONAL verdict; it adds a specific condition (verify the baseline) to the acceptance criteria. The paper deserves credit for full ablations, standard deviations, and an honest discussion of the limits of the unsupervised claim; those strengths do not remove the need for baseline verification.","tokens_in":13179,"tokens_out":5044,"duration_ms":55751,"concrete_test":"Run the official UNION code (released by Lentsch et al.) on nuScenes using the same training split, the same CenterPoint detector settings, and the same evaluation script as VESPA; compare its class-agnostic AP/NDS and 3-class mAP with the reimplemented numbers in Tables 1 and 2. If the official UNION exceeds the reimplemented UNION by more than ~5 AP points, or if the official code is unavailable and the original paper's UNION numbers are not reported, the claimed 12.5-point improvement and the 'state-of-the-art AP' conclusion are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Conclusion, Sec. 6) is state-of-the-art AP for un(human)-supervised object discovery. In Table 1, the only prior VLM+Lidar method in the same family, UNION, is not evaluated with the original authors' released numbers: the footnote states 'We reimplemented UNION based on the original code.' The table reports UNION at 38.4 AP / 31.2 NDS and VESPA at 52.95 / 48.12, i.e., a 12.5-point AP gap. No evidence is given that this reimplementation reproduces the published UNION results: no configuration, no checkpoint, no comparison against original paper values. All other baselines (OYSTER, LISO, CM3D) use published numbers, so UNION is the only one at risk of a different evaluation protocol or undertuning. The 3-class ablation (Table 4) shows that without the cluster-denoising module (DN), VESPA drops to 27.86 mAP, essentially UNION's 27.19 mAP; hence the SOTA margin is carried by modules whose interaction with a possibly stronger baseline is unknown. If the official UNION reaches even ~45 AP, the claimed improvement nearly halves and the SOTA statement may no longer hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VESPA, a multimodal autolabeling pipeline that generates 3D bounding-box pseudolabels for autonomous driving scenes by combining LiDAR point clouds with open-vocabulary vision-language model (VLM) outputs. The pipeline consists of ground removal, Grounding DINO + SAM 2D proposal generation, LiDAR-to-image reprojection, DBSCAN-based cluster denoising with LLM-provided class-width priors, multi-camera merging, DINOv2-based appearance tracking with ICP motion estimation, and LLM-prior-based box refinement. The authors evaluate on nuScenes, reporting class-agnostic AP of 52.95 and NDS of 48.12 (improving on a reimplemented UNION baseline by 12.5 AP points), and a 3-class downstream CenterPoint mAP of 46.54. They also report ablations (Table 4) showing the tracking and cluster-denoising modules have the largest impact, and an additional evaluation on a proprietary dataset.","tokens_in":13457,"tokens_out":7920,"duration_ms":83233,"significance":"If correct, VESPA is a strong autolabeling result: it closes much of the gap to fully supervised detection, and the component ablations are informative. The pipeline is modular and uses publicly available foundation models. However, the state-of-the-art claim is contingent on the fidelity of the reimplemented UNION baseline and on the reproducibility of the LLM-derived parameters; both need to be addressed before the claim can be accepted.","major_comments":[{"comment":"The UNION baseline is a reimplementation, not the original authors' released numbers, and no verification of this reimplementation is provided. Because the paper's central claim of state-of-the-art AP rests on the 12.5 AP point improvement over this reimplemented baseline, the comparison is not yet convincing. Please either report the official UNION results as originally published, or, if a reimplementation is necessary, demonstrate its fidelity by reproducing one or more reported UNION metrics using the original code/checkpoints on the same protocol. Without this, the claimed margin over the prior state of the art is not established.","section":"4.2, Table 1"},{"comment":"The pipeline's critical hyperparameters are not transparent. Section 3.4 states that the DBSCAN distance threshold is 'the average width of the object class' retrieved from an LLM; Section 3.7 uses an LLM-queried maximum velocity; Section 3.8 uses LLM-sourced box-size priors. Section 4.1 says all hyperparameters were 'set by visual tuning on three representative training scenes,' but neither the exact values, the prompts, nor the identities of the three scenes are given. This is load-bearing because the ablation in Table 4 shows that removing the DBSCAN denoising module (DN) reduces 3-class mAP from 46.76 to 27.86, nearly equal to the reimplemented UNION baseline (27.19 in Table 2). The full configuration of these parameters must be disclosed (in the paper or a supplement) to allow reproduction and to substantiate the claim that the LLM priors avoid dataset-specific tuning.","section":"3.4, 3.7, 3.8, 4.1"}],"minor_comments":[{"comment":"In Table 1, the method label 'CMD3D' appears; the text and reference [9] use 'CM3D'. Please correct this typo.","section":"Table 1"},{"comment":"The text reports the effect of removing the tracking module as a '21% drop in mAP' and '22% drop in NDS.' The numbers in Table 4 (46.76 to 25.48 mAP; 43.47 to 21.84 NDS) correspond to drops of about 21 percentage points (and about 45% relative). Please clarify whether you mean percentage points or relative percentages throughout the ablation discussion.","section":"4.3, Table 4"},{"comment":"There is a discrepancy about the split used for pseudo-label quality metrics: Section 4.1 says pseudo labels are compared to ground truth on the validation set, but Table 2's caption says 'pseudo label quality on the training set.' Please align the text and caption.","section":"4.1, Table 2"},{"comment":"The abstract and discussion claim 'open-world' and 'novel category' discovery, but all experiments use the predefined nuScenes classes as queries and evaluate only on those classes. Please provide a demonstration on a class not in the standard set, or soften the claim to 'open-vocabulary queries supported, evaluated on the nuScenes class set.'","section":"Abstract, 5"},{"comment":"For reproducibility, please specify which LLM was used to obtain the priors, the exact prompt templates, and the exact threshold values used for DBSCAN distance, tracking proximity, and box inflation. Also report the Grounding DINO prompt list.","section":"3.2, 3.4, 3.7"},{"comment":"The full VESPA configuration in Table 4 reports mAP of 46.76 and NDS of 43.47, while Table 2 reports 46.54 and 43.45 for the same setting. Please explain whether these are from different runs or a typo; if random variation, state the number of runs.","section":"Table 2, Table 4"}],"recommendation":"major_revision","confidential_remarks":"This is a solid pipeline paper with thorough ablations, but the SOTA claim is premature until the UNION comparison is verified. The missing hyperparameters (especially the LLM-derived DBSCAN threshold) are a serious reproducibility issue for a field that relies on precise configuration. The paper fits the journal's scope and the core method is promising; I would be willing to re-review after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straight to it: this is a genuinely useful engineering contribution, and the experiments are a cut above the usual autolabeling paper. The VESPA pipeline — Grounding DINO/SAM for masks, LiDAR reprojection, DBSCAN denoising, DINOv2 tracking, ICP for velocity — is well thought out, and the ablations actually tell you what each module does. The gains over the reimplemented UNION baseline are large, especially on small objects like bicycles and pedestrians, which is a real weakness of lidar-only methods. The open-vocabulary design and the honest discussion of how VLM supervision is not 'unsupervised' count in its favor.\n\nThe soft spots are real, but mostly fixable. The one that matters is the UNION baseline. The paper reimplements UNION from original code and reports it at 38.4 AP, but gives no evidence that this reproduction matches the original paper's numbers. All other baselines use published scores. If official UNION is stronger, the claimed 12.5-point improvement and the 'state-of-the-art' line in the conclusion are not supported. This is not a fatal flaw — the pipeline itself wouldn't collapse — but it undercuts the headline claim until checked.\n\nSecond, the LLM-queried priors — average class width for DBSCAN, max velocity for tracking, box inflation means — are not reported. These are effectively hyperparameters, and the ablation shows the denoising module that uses the width prior is responsible for a 19-point mAP swing. Without the actual values, a reader cannot reproduce the method.\n\nThird, the hyperparameters were visually tuned on three training scenes and not listed. That is a mild overfitting risk, though they are fixed across all scenes, which is reassuring.\n\nThe paper also gives standard deviations for only one table, leaving the reader to guess at the stability of the 8-class numbers. Minor.\n\nOverall: the core idea is sound, the evaluation is serious, and the paper deserves a proper referee. Before publication I would want the UNION reimplementation details and the LLM priors disclosed, and a direct comparison against official UNION numbers if available. Then the SOTA claim is likely to hold, though perhaps by a smaller margin.\n\nI'd bring this to a reading group focused on autolabeling or VLM-based perception, and I'd cite it if I were writing about 3D pseudo-labeling.","headline":"Solid VLM-guided autolabeling pipeline with strong nuScenes results, but the SOTA claim rests on an unverified UNION reimplementation and undisclosed LLM priors.","tokens_in":14041,"tokens_out":2507,"would_cite":true,"duration_ms":29079,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VESPA claims state-of-the-art unsupervised 3D object discovery on nuScenes, reaching 52.95 class-agnostic AP and 48.12 NDS without human labels.","keywords":["3D object detection","autolabeling","vision-language models","LiDAR point clouds","pseudo-labeling","open-vocabulary detection","autonomous driving","self-supervised learning"],"falsifier":"Run the pipeline on the nuScenes validation set while sweeping the DBSCAN distance threshold over a small range around the quoted class widths (e.g., 0.5 m to 3 m in 0.25 m steps). If the class-agnostic AP moves by more than a few points across that sweep, the denoising step is not the stable mechanism the paper claims; if the paper's threshold values are simply missing, reproducing the reported 52.95 AP requires them.","tokens_in":12963,"feed_emoji":"🚗","tokens_out":11150,"duration_ms":111103,"temperature":0.7,"pith_summary":"VESPA's central claim is that a multimodal pipeline combining LiDAR point clouds with camera images processed by vision-language models can generate high-quality 3D bounding-box pseudolabels for autonomous driving scenes with no human-annotated ground truth. The paper reports that these pseudolabels yield a class-agnostic object-discovery AP of 52.95 and an NDS of 48.12 on the nuScenes validation set, and that training a CenterPoint detector on them reaches 46.54 mAP in a 3-class setting, outperforming prior un(human)-supervised methods. The practical stake is the annotation bottleneck: if VESPA transfers, new scenes, sensor configurations, and even novel object categories can be labeled from off-the-shelf VLMs and LiDAR alone.","feed_headline":"52.95 AP for 3D object discovery with zero human labels","feed_subtitle":"VESPA fuses LiDAR geometry with VLM semantics to beat prior autolabeling pipelines by double-digit points.","key_machinery":"The load-bearing mechanism is a cascade: Grounding DINO proposes open-vocabulary 2D boxes (confidence at least 0.3), SAM converts them to segmentation masks, and LiDAR points are reprojected into each mask using the dataset's extrinsic and ego-pose calibration. The resulting object clusters are denoised by DBSCAN in the XY plane, with its distance threshold set to an average class width retrieved from a large language model rather than tuned on the dataset. Multi-camera proposals are merged, DINOv2 appearance embeddings drive temporal tracking, and ICP provides motion and orientation refinement; final boxes are inflated to LLM-provided class priors. The LLM-queried values (class widths, urban velocity bounds) are what let the pipeline avoid dataset-specific tuning, and the ablations show that the tracking and denoising modules account for the largest performance swings.","core_discovery":"The core discovery is that VLM-based open-vocabulary 2D detections and masks, when reprojected into the LiDAR point cloud and passed through geometric denoising, appearance-based tracking, and motion/box refinement, become reliable 3D object proposals without any ground-truth labels or HD maps. On nuScenes, VESPA outperforms the reimplemented UNION baseline by roughly 14.6 points in class-agnostic AP (52.95 versus 38.4) and reaches about 64% of fully supervised class-agnostic AP, a level the paper situates between training on 5% and 10% of the fully supervised data. The authors state that VESPA's classification capacity is inherited from the VLM, making the pseudolabels open-world to the extent the VLM is, and they frame the work as a step toward eliminating manual supervision while acknowledging that VLM distillation is itself a form of supervision.","pith_inferences":["One reproducibility gap the paper leaves open is that the LLM-queried class widths that set the DBSCAN threshold are not reported; publishing them and testing a threshold sweep would show whether the denoising step's 19% contribution is stable or tuned to a lucky value.","Because classification is inherited from the VLM, a natural test of the 'open-world' claim is to query with categories absent from nuScenes classes (road debris, animals, trailers without tractors) and see whether the pipeline produces coherent 3D pseudolabels for them.","The resolution sensitivity suggests a concrete next experiment: running the same pipeline with higher-resolution surround cameras (or super-resolution preprocessing) should narrow the gap to fully supervised detection more than adding LiDAR channels did, a prediction the paper's data already hints at but does not test."],"forward_implications":["A CenterPoint detector trained on VESPA pseudolabels reaches 46.54 mAP in the 3-class setting, more than double the 27.19 mAP of the same detector on UNION pseudolabels.","VESPA discovers small and rare classes (bicycles, motorcycles, construction vehicles) that prior LiDAR-based autolabelers miss entirely, suggesting the image semantics recover objects whose LiDAR signature alone is too sparse.","The pipeline runs on a different sensor setup (AnonymousScenes) with no retuning, although low-resolution surround cameras degrade performance, indicating image resolution is the current ceiling.","Swapping the VLM from Grounding DINO to OWLv2 changes results only modestly, so future improvements in VLMs should transfer directly into better pseudolabels without pipeline changes.","Removing the tracking module drops mAP by 21% and removing cluster denoising drops it by up to 19%, so the advertised results depend on the geometric and temporal correction steps, not on the 2D detections alone."],"supporting_citations":[{"why":"Grounding DINO supplies the open-vocabulary 2D detections that seed the pipeline.","marker":"[13]"},{"why":"SAM generates the segmentation masks into which LiDAR points are reprojected.","marker":"[10]"},{"why":"UNION is the reimplemented baseline whose AP and NDS VESPA claims to surpass.","marker":"[12]"},{"why":"nuScenes provides the benchmark, sensor set, and evaluation protocol for all main results.","marker":"[4]"},{"why":"CenterPoint is the detector trained on pseudolabels to measure downstream quality.","marker":"[26]"},{"why":"DBSCAN performs the XY-plane cluster denoising that the ablation shows is critical.","marker":"[5]"},{"why":"DINOv2 appearance embeddings enable multi-camera merging and temporal tracking.","marker":"[18]"},{"why":"ICP rigid alignment supplies the motion estimates used for velocity and orientation refinement.","marker":"[3]"}],"fun_headline_variants":["VESPA labels point clouds in the open world, no humans required","VESPA fuses LiDAR and VLM for 52.95 AP autolabeling","Open-world 3D autolabeling hits 52.95 AP on nuScenes","VESPA beats UNION by 14.6 points with zero labels","VLM-powered VESPA discovers new classes in LiDAR data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"VESPA's results assume that DBSCAN clustering in the XY plane, with a distance threshold set to an average class width queried from a language model, reliably separates true object points from background points projected into foreground masks through occlusion and sensor parallax; the exact threshold values are never reported, so this assumption cannot be checked from the paper.","fun_headline_variants_meta":{"raw":{"variants":["VESPA labels point clouds in the open world, no humans required","VESPA fuses LiDAR and VLM for 52.95 AP autolabeling","Open-world 3D autolabeling hits 52.95 AP on nuScenes","VESPA beats UNION by 14.6 points with zero labels","VLM-powered VESPA discovers new classes in LiDAR data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1818,"prompt_tokens":981,"completion_tokens":837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":597,"tokens_out":837,"duration_ms":9847,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:33:41.754109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on the nuScenes validation set while sweeping the DBSCAN distance threshold over a small range around the quoted class widths (e.g., 0.5 m to 3 m in 0.25 m steps). If the class-agnostic AP moves by more than a few points across that sweep, the denoising step is not the stable mechanism the paper claims; if the paper's threshold values are simply missing, reproducing the reported 52.95 AP requires them.","supporting_citations":[{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Grounding DINO supplies the open-vocabulary 2D detections that seed the pipeline."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM generates the segmentation masks into which LiDAR points are reprojected."},{"cited_title":"Union: Unsupervised 3d object detection using object appearance- based pseudo-classes","cited_arxiv_id":null,"evidence_quote":"UNION is the reimplemented baseline whose AP and NDS VESPA claims to surpass."},{"cited_title":"nuscenes: A multi- modal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"nuScenes provides the benchmark, sensor set, and evaluation protocol for all main results."},{"cited_title":"Center- based 3d object detection and tracking","cited_arxiv_id":null,"evidence_quote":"CenterPoint is the detector trained on pseudolabels to measure downstream quality."},{"cited_title":"Method for registration of 3-d shapes","cited_arxiv_id":null,"evidence_quote":"ICP rigid alignment supplies the motion estimates used for velocity and orientation refinement."}],"review_version":1}