{"id":"056cdf6b-d747-4d34-9ca0-57dc7f131762","arxiv_id":"2509.10842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A pipeline that renders raw urban point clouds into virtual multi-view images, extracts mask-level vision-language features, and distills them into a 3D backbone, enabling open-vocabulary text-query segmentation without aligned images or annotations.","lead":"OpenUrban3D labels large urban 3D point clouds with open-vocabulary categories using only the point cloud itself, no aligned photos or manual labels. It renders the scene from many virtual viewpoints, reads mask-level features from a vision-language model, and distills them into a 3D network for zero-shot text queries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline mIoU depends on an undisclosed 'Processed Prompt'; with original class names SensatUrban drops from 39.6% to 30.7%, so the open-vocabulary claim is not yet reproducible.","rationale":"The reader's weakest assumption correctly identifies the prompt sensitivity as the central vulnerability. The paper's own Table IV provides direct evidence: the headline SensatUrban number changes by 8.9 mIoU (from 30.7% to 39.6%) solely from changing the text prompt, and the processed prompt text is never disclosed. This matters because the paper claims zero-shot segmentation for arbitrary natural language and superiority over open-vocabulary baselines; if the main results require carefully engineered per-class prompts, the comparison is not a fair test of open-vocabulary capability. The paper also leaves K, A, R, and alpha unspecified for the main experiments, even though Table V shows non-trivial sensitivity to these settings. Together, these omissions make the strongest claim non-reproducible and not safely attributable to the proposed architecture. I agree with the reader's CONDITIONAL verdict: the framework is plausible and the honest prompt ablation is a point in its favor, but the evidence as presented does not substantiate the headline until the prompts and hyperparameters are released and the zero-shot behavior is verified on both datasets. I therefore recommend no change to the reader's verdict.","tokens_in":18021,"tokens_out":3106,"duration_ms":27666,"concrete_test":"Obtain from the authors the exact 'Processed Prompt' text for every class on SensatUrban and SUM, together with the K, A, R, and alpha settings used in Tables I and II. Then rerun the SUM evaluation twice: once with the original class names and once with the processed prompts, mirroring the SensatUrban ablation in Table IV. If the SUM mIoU drops materially below 75.4%, or if the prompts cannot be provided, the headline result is a tuned, non-reproducible configuration rather than a zero-shot open-vocabulary system. As a control, also evaluate OpenScene under the same processed prompts; if its mIoU rises substantially, the claimed gap is partly prompt leverage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the reported performance is achieved by the proposed framework as an open-vocabulary system, i.e., from arbitrary text queries. Table IV shows this is not the case: on SensatUrban, using the original class-name prompt yields 30.7% mIoU and 72.5% OA, while the 'Processed Prompt' yields the headline 39.6% mIoU and 84.7% OA. The exact processed prompt strings are not given anywhere in the paper, and the SUM table reports only the processed-prompt result. Because the framework's only text interface is cosine similarity to CLIP text embeddings, a 9-point mIoU swing from prompt wording means the central comparison against baselines is a comparison of tuned prompts, not of zero-shot open-vocabulary capability. The paper also does not report the specific values of K, A, R, and alpha used for the main results, despite Table V showing mIoU variation from 72.3% to 77.2% on SUM with these settings. Without the prompts and hyperparameter values, the headline numbers cannot be reproduced or attributed to the architecture rather than to prompt engineering. This is not an internal inconsistency, but it is a missing-evidence problem for the paper's strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OpenUrban3D, a framework for open-vocabulary semantic segmentation of large-scale urban point clouds that uses only raw geometry, without aligned RGB images, pre-trained 3D segmentors, or manual annotations. Virtual multi-view, multi-granularity images are rendered from the point cloud; a frozen vision-language model (ODISE image encoder with CLIP text encoder) extracts mask-level features that are back-projected and fused with a sample-balanced strategy; these 2D features are distilled into a MinkUNet 3D backbone, and at inference the 2D and 3D features are weight-averaged and compared with CLIP text embeddings. A large language model can optionally parse free-form user queries into category lists. Experiments on SensatUrban and SUM report mIoU of 39.6% and 75.4%, respectively, outperforming open-vocabulary baselines and, on SUM, the fully supervised PTV3. Ablations study the projection hyperparameters, sample-balanced fusion, fusion strategy, fusion weight, and text prompts.","tokens_in":18325,"tokens_out":4943,"duration_ms":42366,"significance":"If the reported results are reproducible, OpenUrban3D would be a practically valuable contribution: it is the first 3D open-vocabulary segmentation pipeline for large-scale urban point clouds that does not require aligned imagery or 3D annotations, and its cross-scene generalization claim is well motivated. The paper also ships a detailed architectural description and several ablation studies, which is commendable. However, the central quantitative claims are not yet fully supported: the headline results depend on undisclosed hand-crafted text prompts and unreported hyperparameter values, and there is an apparent duplication of baseline numbers across the two benchmark tables. These issues are fixable within the manuscript's scope, but they must be addressed before the performance claims can be accepted.","major_comments":[{"comment":"The fully supervised baseline rows for PointNet, PointNet++, SPGraph, and SparseConv are numerically identical in Table I (SensatUrban) and Table II (SUM): the same mIoU values (23.7, 32.9, 37.2, 42.6), the same OA values (80.7, 84.3, 76.9, 85.2), and the same per-class IoU entries appear in both tables even though the two datasets have different class sets (13 classes on SensatUrban versus 7 classes on SUM). This duplication is implausible and undermines the credibility of the comparison against fully supervised methods, including the claim that OpenUrban3D surpasses PTV3 on SUM. Please correct the tables, explain the discrepancy, or provide the original baseline log files for verification.","section":"IV-B, Tables I and II"},{"comment":"The headline SensatUrban result of 39.6% mIoU is obtained only with a 'Processed Prompt' that is never specified anywhere in the paper. Table IV shows that using the original class-name prompt drops mIoU to 30.7%. Since the model's only text interface is cosine similarity against CLIP text embeddings, this 9-point mIoU swing means the main comparison is between a tuned prompt and the baselines, not a reproducible zero-shot prediction from arbitrary natural language. Please provide the exact processed prompt strings for both SensatUrban and SUM, report results for both the original and processed prompts on both datasets, and state clearly how these prompts were selected; ideally, prompt selection should not use the test-set metrics directly.","section":"IV-D.5, Table IV"},{"comment":"The specific values of K, A, R, and the fusion weight alpha used for the main results in Tables I, II, and IV are not reported. Section IV-D.1 and Table V show that mIoU on SUM varies from 72.3% to 77.2% as K, A, and R change, and Figures 11 and 12 show a clear performance peak in alpha, but the text only says the optimum is 'near 0.1' without giving the exact value. Without these numbers, the reported results cannot be reproduced or attributed to the architecture rather than to the chosen operating point. Please report the exact K, A, R, and alpha used for every result table, and consider reporting a small sensitivity range for alpha.","section":"IV-D.1 and Figures 11/12"},{"comment":"There is an inconsistency in which points are evaluated in different tables. The 'Fusion' rows in Table VI (SensatUrban 40.1 mIoU, SUM 78.5 mIoU) are numerically identical to the 'OpenUrban3D (2D-3D vis)' rows in Table III, indicating that Table VI evaluates only the 2D-visible subset, but this is not stated. The complete-model numbers in Table III are 39.6% mIoU on SensatUrban and 75.4% mIoU on SUM, which differ by 1.5 and 3.1 points respectively. This ambiguity directly affects the comparison with PTV3 (74.0% mIoU) and must be clarified: state explicitly which point subset is used in each table and report the full-scene numbers consistently in all ablation tables.","section":"IV-C and IV-D.3, Tables III and VI"}],"minor_comments":[{"comment":"The example output in Section III.G reads '[sidewall, road]'; the intended class is presumably 'sidewalk', and the typo should be corrected.","section":"III.G"},{"comment":"In the introduction, 'an 3D open-vocabulary' should be 'a 3D open-vocabulary'.","section":"I"},{"comment":"The caption of Table V does not state the dataset or the feature combination used for the reported mIoU values; specify that these are SUM results and whether they use the full 2D-3D fused features or a different configuration.","section":"IV-D.1, Table V caption"},{"comment":"The LLM-powered text query module is described in Section III.G but never evaluated. At least one quantitative or qualitative example of a free-form LLM-parsed query should be included to substantiate the claimed 'arbitrary natural language' interface.","section":"III.G and IV"},{"comment":"In the SBFF ablation, the mIoU gain is only 0.3 percentage points (27.9% to 28.2%); describing this as 'compelling evidence' is overstated. The per-class gains are meaningful, but the wording should match the modest overall improvement.","section":"IV-D.2"},{"comment":"The related-work claim that Suzuki et al. is 'impractical for occlusion-heavy urban scenes' is cited to a paper on human part segmentation; a more directly relevant urban-scene reference would strengthen the argument.","section":"II.C"}],"recommendation":"major_revision","confidential_remarks":"The duplicated baseline numbers in Tables I and II are the most serious red flag; they suggest a data-handling or transcription error rather than deliberate misconduct, but the authors must be asked to supply original baseline runs or a clear explanation. The undisclosed processed prompts and hyperparameters are also a reproducibility problem that should be fixed before publication. The paper's overall idea is interesting and the ablations are useful, so I would not reject it, but the quantitative claims need to be made fully transparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a genuinely useful idea — virtual rendering plus ODISE masks plus distillation gives you open-vocab segmentation on raw urban point clouds without aligned images — but the paper as written does not let you reproduce or fully trust the headline numbers. It deserves a serious referee, not a desk reject, but the referee should demand the missing details.\n\nWhat's new: the multi-view multi-granularity projection adapted to city scale, the sample-balanced fusion to handle the extreme class imbalance in urban data, and the hybrid 2D-3D feature fusion at inference. The qualitative results look credible, and the ablation on prompt wording is actually honest — it shows the method is sensitive to prompt design, which is important. The paper is also clearly written and the limitations section is candid.\n\nThe soft spots are real and they cluster around evidence. First, Tables I and II list identical baseline rows (PointNet, PointNet++, SPGraph, SparseConv) for two datasets with different class sets and distributions. That's either a copy-paste error or a wrong baseline; either way it has to be fixed. Second, the headline SensatUrban result (39.6 mIoU) uses a 'Processed Prompt' that is never given. The same model with the original class-name prompt drops to 30.7. That's a nine-point swing from prompt text, and it means the comparison against OpenScene, PLA, and RegionPLC is partly a comparison of prompt engineering. The SUM results don't report a prompt ablation at all. Third, the projection hyperparameters (K, A, R) and the fusion weight alpha are not reported for the main runs, even though Table V shows mIoU varying from 72.3 to 77.2 with those settings. Fourth, the abstract claims improved cross-scene generalization, but there is no cross-dataset experiment.\n\nNone of these are fatal to the core idea, and the method is not internally incoherent. But the paper is not yet reproducible. My recommendation: send it to peer review, but with a clear request for the prompt templates, the exact hyperparameter settings, corrected baseline tables, and ideally a cross-dataset test. If those come back, the paper could be a solid contribution. As is, I'd cite it only with caution.","headline":"A plausible annotation-free open-vocab pipeline for urban point clouds, but the headline numbers rest on undisclosed prompts and duplicated baseline tables; fixable, but not yet reliable.","tokens_in":18826,"tokens_out":2533,"would_cite":false,"duration_ms":22034,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenUrban3D claims the first open-vocabulary semantic segmentation of large-scale urban point clouds from raw geometry alone, with no aligned images, pre-trained 3D segmentors, or manual labels.","keywords":["open-vocabulary semantic segmentation","urban point clouds","knowledge distillation","vision-language models","multi-view projection","zero-shot segmentation","SensatUrban","SUM"],"falsifier":"Run OpenUrban3D on SensatUrban using only the benchmark's original class names as queries (e.g., 'wall', 'rail', 'bike') and compare the resulting mIoU against the paper's 39.6%; the paper's own ablation predicts about 30.7%, so a replication landing near the lower figure, or showing large mIoU swings under small prompt paraphrases, would falsify the claim of robust open-vocabulary generalization.","tokens_in":17836,"feed_emoji":"🏙️","tokens_out":8296,"duration_ms":64437,"temperature":0.7,"pith_summary":"The paper sets out to show that open-vocabulary semantic segmentation of large-scale urban point clouds can be achieved from raw geometry alone, with no aligned photographs, no pre-trained 3D segmentation networks, and no manual labels. Its pipeline, OpenUrban3D, renders the point cloud from dozens of virtual viewpoints, extracts mask-level vision-language features from those renderings, fuses them with a sample-balanced scheme, and distills them into a 3D backbone that can be queried by arbitrary text at inference. The stakes are practical: urban scans are plentiful, while aligned imagery and dense annotations are scarce, so a method that removes that dependency would let planners, disaster-response teams, and digital-twin systems query raw scans in natural language. The paper reports 39.6% mIoU on SensatUrban and 75.4% mIoU on SUM, surpassing open-vocabulary baselines and, on SUM, the fully supervised PTV3.","feed_headline":"Text queries segment city-scale point clouds with no labels","feed_subtitle":"OpenUrban3D renders raw LiDAR, distills vision-language features, and beats a supervised model on Helsinki's SUM benchmark.","key_machinery":"The load-bearing mechanism is a multi-view, multi-granularity projection module that places global and local virtual cameras around the scene's bounding box, renders the point cloud into RGB images with depth maps, and feeds those renders to a frozen mask-based vision-language model (ODISE). Mask features are back-projected onto 3D points using pinhole camera geometry plus Z-buffer depth validation, then fused across views by sample-balanced fusion, which down-samples over-represented large masks to prevent the distillation from being dominated by big objects. The fused features act as a teacher for a MinkUNet 3D backbone trained by minimizing $1 - \\cos(F^{3D}, \\text{stopgrad}(F^{2D}))$, aligning 3D features with CLIP's text space. At inference, the fused feature is $F^{fusion} = \\alpha F^{3D} + (1-\\alpha) F^{2D}$ with $\\alpha$ near 0.1. An optional LLM first parses free-form commands into concrete class names, which are then embedded by CLIP.","core_discovery":"On its own terms, OpenUrban3D's central discovery is that a 'segment-first, recognize-later' pipeline, which relies on pre-trained 3D instance segmentors and aligned RGB sequences, can be replaced by a rendering-and-distillation loop that works with raw point clouds only. The paper claims that rendering the point cloud from multi-scale virtual cameras, back-projecting mask-level vision-language features with Z-buffer occlusion checks, and distilling them into a MinkUNet backbone yields 3D features aligned with CLIP text space. At inference, a weighted fusion of 2D and 3D features, with the 3D weight kept small, assigns each point the class whose text embedding has the highest cosine similarity. The reported results are 39.6% mIoU and 84.7% OA on SensatUrban, and 75.4% mIoU and 90.5% OA on SUM, with the SUM figure exceeding the fully supervised PTV3's 74.0% mIoU.","pith_inferences":["A testable extension is automatic prompt optimization: if an LLM proposes candidate class descriptions and a small labeled probe set (or the model's own confidence) scores them, much of the 30.7-to-39.6 prompt gap might be recovered without manual tuning.","The small gap between 2D-visible mIoU (40.1) and full-scene mIoU (39.6) on SensatUrban suggests the distilled 3D branch carries occlusion-covered points well; pushing the same pipeline into denser street canyons or under bridges would test that robustness harder.","Because the method renders colorized point clouds, scenes with poor or missing colorization (night scans, shadow-heavy alleyways, sensor failure) may see degraded distillation; replacing color with geometric cues such as normals or height could make the pipeline color-agnostic.","The method's reliance on rendered RGB means its ceiling is tied to rendering quality and to the frozen 2D feature extractor; coupling it with diffusion-based or neural rendering might improve mask quality on low-density scans."],"forward_implications":["Large-scale urban datasets that have discarded their original aerial or mobile images become usable for open-vocabulary segmentation, since only the point cloud is needed.","On SUM, the method surpasses the fully supervised PTV3, suggesting that web-scale vision-language knowledge distilled into a 3D backbone can compete with dense manual labels when categories are well defined.","The same trained model can be queried with novel text concepts at inference without retraining, since prediction is a cosine similarity against CLIP text embeddings.","The fusion study indicates 3D geometry's main contribution is boundary refinement rather than semantic content, which implies hybrid 2D-semantic and 3D-geometric designs are the right direction for urban point cloud understanding.","The prompt ablation shows performance swings by roughly 9 mIoU points between original and processed prompts, so open-vocabulary accuracy in practice depends on prompt precision as much as on the model."],"supporting_citations":[{"why":"Supplies the frozen mask-based 2D vision-language model whose object-level features are extracted from every rendered view and back-projected to points.","marker":"[18]"},{"why":"Provides the distillation paradigm of using 2D features as a teacher to align a 3D encoder with CLIP's text space.","marker":"[19]"},{"why":"Supplies the pre-trained text encoder used to score fused point features against class-name embeddings at inference.","marker":"[33]"},{"why":"The UAV photogrammetry benchmark on which OpenUrban3D is trained without labels and evaluated; its class definitions structure the SensatUrban comparison.","marker":"[22]"},{"why":"The Helsinki mesh-based benchmark whose 30 points/m2 version is used for training and evaluation; the comparison to fully supervised PTV3 happens on this dataset.","marker":"[23]"},{"why":"The 3D backbone (MinkUNet) that receives the distilled 2D features and produces the per-point 3D features used in fusion.","marker":"[30]"},{"why":"PTV3 is the fully supervised state-of-the-art baseline on SUM that OpenUrban3D reports surpassing (74.0% vs 75.4% mIoU).","marker":"[12]"}],"fun_headline_variants":["Text prompts segment city LiDAR without labels or RGB","OpenUrban3D: zero-shot open-vocabulary segmentation from raw LiDAR","No annotations, no RGB: open-vocabulary LiDAR segmentation","OpenUrban3D beats supervised baseline on SUM, no labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers depend on a manually tuned 'Processed Prompt' for each benchmark class whose exact wording is not given; with the original class-name prompts, reported SensatUrban mIoU drops from 39.6% to 30.7%.","fun_headline_variants_meta":{"raw":{"variants":["Text prompts segment city LiDAR without labels or RGB","OpenUrban3D: zero-shot open-vocabulary segmentation from raw LiDAR","No annotations, no RGB: open-vocabulary LiDAR segmentation","OpenUrban3D beats supervised baseline on SUM, no labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001106,"raw_usage":{"total_tokens":4652,"prompt_tokens":1028,"completion_tokens":3624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":3550}},"tokens_in":644,"tokens_out":3624,"duration_ms":22252,"temperature":1.0,"reasoning_tokens":3550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:52:41.716322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OpenUrban3D on SensatUrban using only the benchmark's original class names as queries (e.g., 'wall', 'rail', 'bike') and compare the resulting mIoU against the paper's 39.6%; the paper's own ablation predicts about 30.7%, so a replication landing near the lower figure, or showing large mIoU swings under small prompt paraphrases, would falsify the claim of robust open-vocabulary generalization.","supporting_citations":[{"cited_title":"Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges,","cited_arxiv_id":null,"evidence_quote":"The UAV photogrammetry benchmark on which OpenUrban3D is trained without labels and evaluated; its class definitions structure the SensatUrban comparison."},{"cited_title":"Sum: A benchmark dataset of semantic urban meshes,","cited_arxiv_id":null,"evidence_quote":"The Helsinki mesh-based benchmark whose 30 points/m2 version is used for training and evaluation; the comparison to fully supervised PTV3 happens on this dataset."}],"review_version":2}