{"id":"d5996d95-ec84-449f-8966-c6254e416e9e","arxiv_id":"2411.17761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OpenAD introduces the first real-world 3D open-world object detection benchmark for autonomous driving, with 2,000 scenes, 206 corner-case categories, and semantic-aware evaluation metrics.","lead":"OpenAD is a new benchmark for 3D object detection in autonomous driving, built from 2,000 scenes across five existing datasets and annotated with 206 classes of rare or unusual objects. It provides unified 2D and 3D bounding boxes plus evaluation metrics that use semantic similarity, and it includes a vision-centric baseline and an ensemble method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth semantic label accuracy is unverified: the ~90% MLLM figure comes from 30 samples, and final post-correction labels have no measured inter-annotator agreement.","rationale":"The Reader's weakest assumption is annotation accuracy, and my stress-test converges on the same point: the benchmark's value depends on ground-truth semantic labels being correct enough to rank detectors, especially for rare corner-case categories. The evidence provided in the paper is a 30-sample manual check reported as roughly 90% accuracy, which is too small to establish per-category reliability, and the paper simultaneously claims manual verification at every stage without quantifying the final corrected labels. These two statements are in tension: if every annotation was manually verified, the 90% figure describes the raw MLLM, not the benchmark; if no such verification was applied to semantic labels, then 10% of labels could be wrong. Either way, the released benchmark lacks a measured quality guarantee. The withheld labels (Appendix C) make this impossible for users to audit. I considered other concerns—such as the unspecified CLIP variant used for semantic matching and the relatively small number of corner-case instances—but those do not strike at the core claim as directly as annotation reliability. The CLIP model can be documented later without changing the dataset; the small instance count is a limitation but not a correctness issue. The recommended condition is therefore unchanged: the paper should provide a labeled validation subset, inter-annotator agreement statistics, and a per-category error analysis. This does not reject the benchmark, which is likely useful, but it does mean full acceptance should wait for that evidence.","tokens_in":958,"tokens_out":867,"duration_ms":60354,"concrete_test":"Release a held-out validation subset of OpenAD (e.g., 200 scenes with full labels) and independently re-annotate a random sample of at least 300 objects stratified by category frequency, using two annotators per object. Report per-category semantic-label agreement between the released labels and the independent annotations, inter-annotator agreement (e.g., Cohen's kappa), and 3D box agreement. Then recompute OpenAD AP/AR rankings on this subset using the independent labels; if rare-category semantic agreement is below approximately 95% or the evaluated model rankings change materially, the benchmark ground truth needs correction before the rankings can be trusted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that OpenAD provides reliable open-world ground truth for 3D detection—rests on the correctness of the semantic labels and the per-dataset seen/unseen flags. Section 4 and Appendix A report approximately 90% accuracy for the MLLM labeler, but this was measured by manually checking five repetitions on 30 hard samples, not on the final released annotations. The same section also states that manual verification was applied at each stage, so it is unclear whether the 90% refers to raw MLLM output or to final labels after correction; no post-correction accuracy or inter-annotator agreement is reported. Because Appendix C withholds the full label set, an external check of the 206-category taxonomy and the seen/unseen labels is impossible. If MLLM errors are concentrated in rare corner-case categories—precisely the categories OpenAD is designed to evaluate—the semantic true-positive criterion in Section 3.2 and the AR seen/unseen splits in Tables 2 and 3 become biased, and model rankings may shift. This weakens the benchmark's primary contribution unless annotation reliability for rare categories is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OpenAD, a real-world open-world autonomous driving benchmark for 3D object detection. OpenAD contains 2,000 scenes drawn from five existing datasets (Argoverse 2, KITTI, nuScenes, ONCE, Waymo), with 6,597 newly annotated corner-case objects plus 13,164 objects taken from the original dataset annotations, for 19,761 objects in total. Objects are labeled with 2D and 3D bounding boxes, natural-language semantic tags organized into 206 categories, and per-dataset seen/unseen flags. The paper also proposes a semantic-aware evaluation protocol using CLIP-based semantic similarity with positional thresholds, a vision-centric 3D open-ended detection baseline that lifts 2D open-world proposals to 3D boxes, and an ensemble method that fuses open-world and specialized detectors. Extensive experiments compare 2D and 3D open-world and specialized models on the benchmark, and an online challenge is hosted on EvalAI.","tokens_in":15661,"tokens_out":5113,"duration_ms":52906,"significance":"If the annotation quality holds, OpenAD fills a real gap: it is the first real-world benchmark that evaluates 3D open-world object detection in autonomous driving with both corner-case categories and domain-transfer evaluation across multiple datasets. The unified data format, the natural-language semantic labels, and the seen/unseen split are useful innovations, and the paper ships a toolkit, evaluation code, and a public challenge. The baseline analysis is also valuable: it quantifies the complementarity of specialized and open-world models and shows that in-domain gains can come at the cost of open-world capability. However, the benchmark's central value depends on the reliability of its ground-truth semantic labels and seen/unseen flags, and that reliability is not yet convincingly established.","major_comments":[{"comment":"The paper claims in Section 4 that the final MLLM and prompt achieve approximately 90% accuracy on the entire OpenAD dataset, but Appendix A states that accuracy was manually calculated from five repetitions on 30 highly challenging samples. It is also unclear whether the 90% refers to raw MLLM output or to the final labels after the manual correction stage. Because OpenAD's primary contribution is as ground truth for rare corner-case objects, this extrapolation is too thin to establish benchmark reliability. Please report a stratified accuracy estimate on the final released annotations, per-category accuracy for rare categories, and inter-annotator agreement (or an equivalent dual-annotation study), and clarify the role of manual correction in the reported figure.","section":"Section 4 and Appendix A"},{"comment":"The true-positive criterion in the evaluation protocol uses CLIP cosine similarity thresholds (0.5, 0.7, 0.9) for semantic matching, but no calibration or human-agreement analysis is provided to show that these thresholds correctly separate synonymous from incorrect labels for the 206 corner-case categories. Since several evaluated methods (GLIP, OWL-ViT v2, GroundingDINO, YOLO-World) are CLIP-based, the metric may inadvertently favor methods whose embeddings align with CLIP. Please validate the semantic matching with human judgments or an independent text-similarity measure and report how model rankings change across semantic thresholds.","section":"Section 3.2"},{"comment":"The AR-seen and AR-unseen splits in Tables 2 and 3 are central evidence for the paper's domain-generalization and open-endedness conclusions, but Appendix C withholds the full category labels and the underlying seen/unseen flags. These flags are produced by the same MLLM pipeline whose accuracy is only estimated on 30 samples, so errors concentrated in rare corner-case categories could directly bias the seen/unseen comparisons. Please release a public validation subset with full labels and flags, or otherwise provide a detailed audit of the seen/unseen flag-generation step, so that the benchmark's core evaluation can be independently checked.","section":"Appendix C and Tables 2/3"}],"minor_comments":[{"comment":"There are several typographical errors in the table headers and method names, including 'V ovNetv2-99' (should be 'VoVNetv2-99'), 'ARnuscseen', and 'ARothersseen'; these should be corrected in the camera-ready version.","section":"Tables 2 and 3"},{"comment":"Figure 6 is difficult to read because the font sizes for the prompt variants and per-model accuracy rows are very small; please enlarge the figure or restructure it so that the ablation results are legible.","section":"Appendix A, Figure 6"},{"comment":"The statement that each of the 2,000 scenes 'contains at least one corner case object that the original dataset has not annotated' should be reconciled with Figure 2's scene counts by source; please clarify whether the listed counts are before or after the manual scene filtering stage.","section":"Section 3.1"},{"comment":"The metric section does not specify how predictions with multiple or free-form labels are handled when computing CLIP semantic similarity, nor how the 'top 300 predictions' limit is applied to methods that output thousands of low-confidence boxes; a sentence clarifying this would improve reproducibility.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of a datasets-and-benchmarks venue and the resource is potentially valuable. The main risk is not novelty but the reliability of the ground-truth labels and seen/unseen flags, which are the load-bearing components of the benchmark. I would not require full release of all labels before acceptance, but a public validation subset with full labels and a thorough annotation-accuracy study is necessary. If the authors provide that, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OpenAD is worth taking seriously. It is the first real-world 3D benchmark for open-world autonomous driving detection at this scale: 2,000 scenes from five datasets, 206 corner-case categories, and 19.8K objects with 2D/3D boxes plus natural-language labels. The seen/unseen per-dataset splits and the semantic-aware AP/AR protocol address a real gap in the literature. The vision-centric baseline and the general-plus-specialist fusion results are useful, and the observation that some specialized 3D models overfit in-domain and degrade out-of-domain is a concrete finding that in-domain benchmarks miss.\n\nThe soft spots are real but not fatal. The ~90% MLLM annotation accuracy is extrapolated from 30 hard samples tested five times, and the paper never reports post-correction accuracy or inter-annotator agreement. Since the full label set is withheld in Appendix C, an outside check of the 206-class taxonomy and the seen/unseen flags is impossible. This matters because the benchmark's value depends on ground truth being correct in the long tail, exactly where MLLM errors could concentrate. The revision should release a held-out validation subset with labels, report per-category accuracy for rare classes, and specify the CLIP variant used for semantic scoring. None of this makes the benchmark unusable, but the current evidence is weaker than the claims.\n\nThe internal numbers check out—object counts, scene counts, and the seen/unseen splits are consistent. The baseline ablations are reasonable, and the paper is honest about limitations. I do not see invented entities or circularity. I would send this to a serious referee: dataset papers of this scale deserve careful review, and the annotation-quality question needs airing. I would not accept the claims about the fully annotated dataset as-is; I would require the validation-set release and annotation QA statistics before acceptance.","headline":"OpenAD is a genuinely useful 3D open-world detection benchmark, but the annotation-reliability evidence is thinner than the claims require.","tokens_in":16260,"tokens_out":2613,"would_cite":true,"duration_ms":23334,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenAD: a real-world benchmark for 3D detection of rare road objects.","keywords":["open-world object detection","autonomous driving benchmark","3D object detection","corner cases","multimodal large language model","domain generalization","semantic-aware evaluation","2D-to-3D lifting"],"falsifier":"Take a random sample of, say, 200 corner-case objects spanning rare categories, have independent human annotators label them with the same protocol, and compare against OpenAD's MLLM-generated tags. If per-category agreement falls well below the claimed 90% for rare classes, or if swapping in the corrected labels changes the relative ranking of top models, the benchmark's core assumption of trustworthy ground truth is violated.","tokens_in":15227,"feed_emoji":"🚗","tokens_out":4142,"duration_ms":35038,"temperature":0.7,"pith_summary":"The paper introduces OpenAD, a real-world benchmark for open-world 3D object detection in autonomous driving. It selects 2,000 scenes from five existing driving datasets and re-annotates rare 'corner case' objects with natural-language labels spanning 206 categories, giving 19,761 total 2D/3D bounding boxes. The paper also proposes a semantic-aware evaluation protocol that matches predictions to ground truth by both geometry and semantic similarity, so synonymous descriptions count as correct and long-tailed categories do not skew the metric. On top of the benchmark, the authors propose a vision-centric 3D open-world detector baseline that lifts 2D open-world proposals to 3D boxes with a lightweight converter, plus an ensemble method that fuses open-world and specialized models. If the benchmark is sound, it gives the field a common yardstick for measuring whether detectors can find and describe uncommon objects, not just common categories.","feed_headline":"New 3D benchmark tests self-driving detectors on rare road objects","feed_subtitle":"OpenAD unifies five driving datasets, labels 206 corner-case categories, and scores models on objects they never saw in training.","key_machinery":"The load-bearing mechanism is the unified annotation and evaluation pipeline built on a multimodal large language model (MLLM). An MLLM abnormal filter flags scenes with corner-case objects, a second MLLM step assigns natural-language descriptions and seen/unseen flags, and geometric boxes are produced by point-cloud clustering plus manual correction. On the evaluation side, the matching rule treats a prediction as a true positive only when it passes both a positional threshold (IoU in 2D, center distance in 3D) and a semantic similarity threshold computed with CLIP features, so synonymous descriptions count as matches and long-tailed categories do not distort the metric. The same pipeline supports the proposed 2D-to-3D BBox Converter baseline, which turns frozen 2D open-world proposals into 3D boxes with a trainable depth network, pseudo point clouds, and a PointNet branch.","core_discovery":"The central claim is that OpenAD is the first real-world open-world benchmark for 3D object detection in autonomous driving, and that it changes what evaluation can reveal. Existing driving benchmarks only test a handful of predefined categories, so a model that excels on them may still miss unusual objects that matter for driving. OpenAD provides 6,597 corner-case objects plus 13,164 common objects across 2,000 temporally consistent scenes, each annotated with 2D and 3D boxes and a semantic tag, plus a per-object seen/unseen flag for each source dataset. The paper further argues that its semantic-aware metrics, which use CLIP cosine similarity and multiple positional thresholds, expose strengths and weaknesses that in-domain benchmarks hide, such as overfitting that trades away open-world recall.","pith_inferences":["Because OpenAD withholds most category labels, future users cannot easily audit per-category annotation quality; releasing a small public seed of rare-category labels would make benchmark quality independently checkable.","The benchmark could be extended beyond detection: the same natural-language labels and seen/unseen splits could support open-world segmentation, occupancy prediction, or tracking evaluation.","The semantic matching via CLIP may favor models whose embeddings align with CLIP; a model with better geometry but different vocabulary could be under-scored, so an alternative semantic matcher trained on driving text would be a useful stress test."],"forward_implications":["Models that look strong on nuScenes-style in-domain benchmarks can be much weaker on OpenAD's unseen categories and out-of-domain scenes, so OpenAD provides a way to detect overfitting that existing benchmarks miss.","Open-world 2D detectors can be turned into 3D open-world detectors with a small trainable converter, without retraining the 2D model.","Fusing a specialized 3D detector with an open-world detector raises both seen and unseen recall, suggesting the two capabilities are complementary rather than exclusive.","The seen/unseen split lets the benchmark attribute performance separately to domain generalization and open-ended recognition."],"supporting_citations":[{"why":"Supplies the Argoverse 2 scenes used in OpenAD.","marker":"[56]"},{"why":"Supplies nuScenes scenes and common-object annotations, and serves as the training set for specialized baselines.","marker":"[8]"},{"why":"Supplies KITTI scenes used in OpenAD.","marker":"[20]"},{"why":"Supplies ONCE scenes used in OpenAD.","marker":"[45]"},{"why":"Supplies Waymo scenes used in OpenAD.","marker":"[53]"},{"why":"Prior real-world corner-case dataset whose LiDAR-based pipeline OpenAD contrasts with its vision-centric approach.","marker":"[33]"},{"why":"Point-cloud clustering algorithm used to generate candidate 3D bounding boxes.","marker":"[6]"},{"why":"InternVL 1.5 selected as the MLLM for semantic annotation after accuracy comparison.","marker":"[13]"},{"why":"YOLO-World used as a frozen 2D open-world proposal generator for the 3D baseline.","marker":"[14]"},{"why":"GenerateU used as a second 2D open-world proposal generator in the baseline and ablations.","marker":"[15]"}],"fun_headline_variants":["First open-world 3D benchmark for self-driving cars","Benchmark exposes self-driving blind spots on rare objects","OpenAD: testing 3D detection on unseen road objects","Self-driving 3D detection benchmark for rare objects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's usefulness rests on the accuracy of the MLLM-produced semantic labels and seen/unseen flags; the paper estimates about 90% accuracy from manual review of only 30 challenging samples, and full category labels are withheld, so errors concentrated in rare categories could bias rankings.","fun_headline_variants_meta":{"raw":{"variants":["First open-world 3D benchmark for self-driving cars","Benchmark exposes self-driving blind spots on rare objects","OpenAD: testing 3D detection on unseen road objects","Self-driving 3D detection benchmark for rare objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2430,"prompt_tokens":895,"completion_tokens":1535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1469}},"tokens_in":511,"tokens_out":1535,"duration_ms":11293,"temperature":1.0,"reasoning_tokens":1469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:36:32.884058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 200 corner-case objects spanning rare categories, have independent human annotators label them with the same protocol, and compare against OpenAD's MLLM-generated tags. If per-category agreement falls well below the claimed 90% for rare classes, or if swapping in the corrected labels changes the relative ranking of top models, the benchmark's core assumption of trustworthy ground truth is violated.","supporting_citations":[{"cited_title":"Argoverse 2: Next generation datasets for self-driving perception and forecasting","cited_arxiv_id":null,"evidence_quote":"Supplies the Argoverse 2 scenes used in OpenAD."},{"cited_title":"Nuscenes: A multimodal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Supplies nuScenes scenes and common-object annotations, and serves as the training set for specialized baselines."},{"cited_title":"One million scenes for autonomous driving: Once dataset","cited_arxiv_id":null,"evidence_quote":"Supplies ONCE scenes used in OpenAD."},{"cited_title":"Fast range image-based segmentation of sparse 3d laser scans for online operation","cited_arxiv_id":null,"evidence_quote":"Point-cloud clustering algorithm used to generate candidate 3D bounding boxes."},{"cited_title":"Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"InternVL 1.5 selected as the MLLM for semantic annotation after accuracy comparison."},{"cited_title":"Yolo-world: Real-time open-vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"YOLO-World used as a frozen 2D open-world proposal generator for the 3D baseline."},{"cited_title":"Generative region-language pretraining for open-ended object detection","cited_arxiv_id":null,"evidence_quote":"GenerateU used as a second 2D open-world proposal generator in the baseline and ablations."}],"review_version":1}