{"id":"ffd16232-7251-4687-a513-2a95546f8c63","arxiv_id":"2504.13590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"HAECcity is the first open-vocabulary panoptic segmentation system for city-scale point clouds trained without any human labels, using CLIP features projected from synthetic images and a superpoint graph MoE backbone.","lead":"This paper presents HAECcity, a 3D scene understanding system that labels city-scale point clouds without hand-annotated training data, using synthetic images and a superpoint graph transformer with mixture-of-experts blocks. It reports the first open-vocabulary panoptic segmentation on the SensatUrban city dataset, though accuracy is still far below supervised baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-vocabulary capability is demonstrated only on preprocessing features, not on the trained model, so the central claim lacks direct evidence.","rationale":"The reader's weakest assumption is pseudo-label quality, which is a real first-order risk and is explicitly conceded in Section 5. My concern is closely related but sharper: even granting that the pseudo-label targets are usable, the paper never shows that HAEC's own predicted vectors support open-vocabulary queries, because the qualitative demonstrations use preprocessing features. The closed-set metrics on SensatUrban and ScanNet give some evidence that the model learns a useful representation, and the improvement over the oracle is encouraging, but those metrics do not test long-tail or novel-text behavior. Since the central claim is specifically about a trained 3D model enabling open-vocabulary understanding, the missing model-level retrieval demonstration is the most load-bearing gap. A positive result on the proposed test would resolve it; a negative result would reduce the contribution to a noisy-label training study. The reader's CONDITIONAL verdict already captures this evidentiary need, so I do not move the verdict.","tokens_in":11148,"tokens_out":4511,"duration_ms":42771,"concrete_test":"Use the trained HAEC model to produce predicted CLIP vectors for the same two SensatUrban scenes used in Figures 1 and 5, then run the identical cosine-similarity queries ('The gate of King's College', 'a red car') on those predicted vectors and threshold them as in the paper. Compare the resulting localization maps to the oracle-feature maps and to ground truth (e.g., hand-checked bounding regions). If the trained model's retrievals do not localize the target objects at comparable or better precision than the oracle, the central open-vocabulary claim is not supported; if they do, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution is a strictly-3D trained model that performs open-vocabulary panoptic segmentation at city scale. Yet Section 4.3 states 'we query the preprocessing derived point cloud features on two scenes from SensatUrban' for the long-tail queries (Figs. 1 and 5). Those retrieval maps therefore test the OpenSeg/MobileCLIP pseudo-label pipeline, not HAEC's predicted CLIP vectors. The only quantitative test of HAEC's semantic output is closed-set mIoU/mAcc on SensatUrban (Table 2: 22.45/30.38), which measures alignment with a fixed set of class text embeddings; it does not exercise open-vocabulary or long-tail behavior. This is not a minor presentational issue: the third contribution is 'the first employment of a strictly 3D processing model based... open-vocabulary panoptic segmentation system onto city-scale point clouds,' and the evidence currently supports at most the first employment of the pseudo-label pipeline, not of the trained model. The paper itself flags the underlying risk in Section 5: 'the most significant limitation to our approach comes from the quality of the synthetic-image-derived CLIP features,' and the preprocessing oracle mIoU of 5.62 confirms the target quality is low. So the load-bearing, unverified step is that HAEC's own learned features retain enough open-vocabulary alignment to answer novel text queries on city-scale scenes. If that step fails, the central claim reduces to 'a 3D model trained on low-quality open-vocabulary targets achieves moderate closed-set mIoU,' which is not the claimed contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HAEC, a superpoint-graph transformer with mixture-of-experts blocks that predicts per-superpoint CLIP feature vectors and instance labels for point clouds, with the goal of open-vocabulary panoptic segmentation at city scale. Training targets are produced by a synthetic pipeline: synthetic RGB images are rendered from the point cloud, filtered with MobileCLIP, processed by OpenSeg, projected back to 3D, and then clustered with spherical k-means and DBSCAN to form pseudo-classes and pseudo-instances. The model is evaluated on ScanNet for panoptic quality and on SensatUrban for semantic segmentation, and qualitative long-tail queries are shown for two SensatUrban scenes. The paper claims the first strictly-3D, non-projection-based open-vocabulary panoptic system for city-scale point clouds, trained without hand annotations.","tokens_in":11404,"tokens_out":3955,"duration_ms":38848,"significance":"If the central claim were fully supported, this would be a valuable demonstration that a single-pass 3D model can be trained on noisy synthetic pseudo-labels and transfer to human-annotated benchmarks at city scale. The paper has genuine strengths: the model outperforms its own pseudo-label oracle on SensatUrban (mIoU 22.45 vs. 5.62), which suggests the learned representation is not merely memorizing pseudo-labels; inference is a single 3D pass without per-scene image projection; and the attempt to scale open-vocabulary understanding to a multi-square-kilometer dataset is timely. However, the evidence as presented does not yet substantiate the headline open-vocabulary capability of the trained model, because the only open-vocabulary demonstrations query preprocessing features rather than HAEC predictions, and the quantitative evaluation is closed-set. The contribution is therefore promising but currently under-supported.","major_comments":[{"comment":"The open-vocabulary queries shown in Figures 1 and 5 are explicitly computed on 'the preprocessing derived point cloud features' rather than on HAEC's predicted CLIP vectors. This means the paper does not demonstrate that the trained model can answer novel text queries; it demonstrates only that the pseudo-label pipeline can sometimes retrieve a long-tail object from the raw OpenSeg features. Since the third stated contribution is 'the first employment of a strictly 3D processing model based ... open-vocabulary panoptic segmentation system onto city-scale point clouds,' this is a load-bearing gap. Please run the same long-tail and class-agnostic text queries on the HAEC-predicted features, report quantitative retrieval metrics (e.g., mAP or IoU at a similarity threshold) for both the preprocessing features and the model predictions, and compare them on the same scenes.","section":"Section 4.3"},{"comment":"The paper itself concedes in Section 5 that 'the most significant limitation to our approach comes from the quality of the synthetic-image-derived CLIP features,' and the preprocessing oracle mIoU on SensatUrban is only 5.62. Because the training targets (OpenSeg features), the spherical k-means pseudo-classes, and the text-embedding evaluation all live in the same CLIP-derived space, the closed-set mIoU of 22.45 could partly reflect learning the pseudo-label distribution rather than open-vocabulary generalization. To support the open-vocabulary claim, please report the model's alignment with held-out text queries that were not used in the pseudo-class clustering, and compare HAEC's retrieval performance against the preprocessing oracle. A quantitative separation between the model and the oracle on unseen categories would directly address this concern.","section":"Section 5 / Table 2"},{"comment":"The novelty claim that this is the first strictly-3D open-vocabulary panoptic system at city scale is asserted rather than demonstrated against cited related work. In particular, OpenGraph is described as an 'open-vocabulary hierarchical 3D graph representation in large-scale outdoor environments' and Search3D is a hierarchical open-vocabulary 3D segmentation method; neither is quantitatively compared on SensatUrban or ScanNet, and Table 1's checkmark pattern for these methods is not explained. Please provide a concrete comparison or a precise argument for why these systems fall outside the claimed scope, and clarify whether the 'first' claim refers to scale, to the absence of projection at inference, to the absence of hand-labeled training data, or to all three.","section":"Section 1 / Table 1"}],"minor_comments":[{"comment":"The text says 'The processing of the partitions proceeds from fine to course partitions'; 'course' should be 'coarse'.","section":"Section 3.2"},{"comment":"The author name 'Fr ´ed´eric Kaplan' contains LaTeX accent artifacts; this should be rendered as 'Frédéric Kaplan' in the final version.","section":"Header / Title Page"},{"comment":"The table columns include both PQ and mIoU/mAcc for ScanNet and SensatUrban, but several rows (e.g., SuperCluster, RandLA-Net, OpenScene) list values in a way that makes it unclear which metric is being reported. Please separate the two benchmarks into distinct tables or add explicit column headers for each metric and each dataset.","section":"Table 2"},{"comment":"The phrase 'strictly 3D processing model based, rather than 2D to 3D projection based' in the contributions is potentially confusing because the pseudo-label pipeline in Section 3.1 is itself a 2D-to-3D projection method. Please clarify that the claim concerns inference-time processing, and that 2D-to-3D projection is used only to generate training targets.","section":"Section 3.1"},{"comment":"The paper does not report values for the key hyperparameters, including the spherical k-means cluster count K, the DBSCAN epsilon and minPts settings, the triplet margin alpha, the MobileCLIP acceptance threshold, the number of MoE experts, or the loss weighting between Lrec and Ltriplet. Please provide these settings, or an appendix with the full training configuration, to make the experiments reproducible.","section":"Section 3.1 / Section 3.2"},{"comment":"The caption for Figure 4 says the OpenSeg bar is 'representative of the preprocessing time for training samples in our approach,' but it is not clear whether this includes the synthetic image rendering time and the MobileCLIP/Depth Anything filtering. Please specify which preprocessing stages are included in each bar.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest result—the model outperforming its pseudo-label oracle—suggests the approach is worth pursuing, but the absence of any evaluation of the trained model's own open-vocabulary behavior makes the headline claim unverifiable as written. I would be supportive of a revised version that adds quantitative open-vocabulary evaluation on HAEC-predicted features, addresses the novelty comparison with OpenGraph and Search3D, and reports the needed hyperparameters. The current manuscript is not ready for acceptance because the central contribution is not directly tested."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuine attempt to move open-vocabulary 3D segmentation to city scale without hand labels. The authors combine a superpoint graph transformer with mixture-of-experts blocks and a pseudo-label pipeline that builds CLIP targets from synthetic renders of the point cloud, then cluster those features into pseudo-classes and instances. The first application to SensatUrban is new. It beats its own preprocessing oracle by a large margin (22.45 vs 5.62 mIoU), which suggests the model learns structure beyond the noisy targets, and the ScanNet results are comparable to OpenScene while using a fraction of the compute.\n\nWhat is good: the pipeline is clearly described, the limitations section is unusually honest, and the efficiency argument is credible. The paper also gives credit where due to SuperCluster, OpenScene, and others. The oracle comparison is a nice touch, and the model's improvement over it is the strongest evidence in the paper.\n\nThe soft spots are real. The long-tail queries in Section 4.3 — the King's College gate and the red car — are run on the preprocessing features, not on the trained model's predicted CLIP vectors. So the headline capability is not directly demonstrated. The closed-set mIoU on SensatUrban does show that the predicted features align with text embeddings for a fixed set of classes, which is some evidence of open-vocabulary behavior, but it does not exercise novel or long-tail queries. A serious revision should add open-vocabulary evaluations using the model's own outputs, ideally with a few long-tail queries on SensatUrban or a held-out set.\n\nOther gaps are more standard: no error bars, some hyperparameters unreported, no code, and the table formatting is confusing. The pseudo-label oracle mIoU of 5.62 shows the target quality is low, which limits the ceiling. The authors acknowledge this in Section 5.\n\nOn balance, the central idea has merit and the evidence, while incomplete, is not misleading. The paper deserves a serious referee; it will need substantial revision to close the evidence gap but the core contribution is worth engaging with.","headline":"A promising self-supervised open-vocabulary pipeline for city-scale point clouds, but the long-tail showcase runs on the preprocessing features rather than the trained model.","tokens_in":12027,"tokens_out":2637,"would_cite":true,"duration_ms":22862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HAEC claims that a city-scale point cloud can be given open-vocabulary panoptic labels—naming and counting arbitrary objects—by a purely 3D model trained with no human annotation.","keywords":["open-vocabulary segmentation","panoptic segmentation","point clouds","superpoint graph","mixture of experts","pseudo-labeling","CLIP features","city-scale 3D understanding"],"falsifier":"Project the pseudo-label features on a held-out SensatUrban scene and test free-text queries such as “a red car” against manually verified regions; if retrieved points show no significant cosine-similarity advantage over random regions, or if the preprocessing oracle mIoU is within noise of the majority-class baseline, the pipeline cannot supply usable training targets.","tokens_in":10861,"feed_emoji":"🏙️","tokens_out":5079,"duration_ms":42742,"temperature":0.7,"pith_summary":"This paper claims that open-vocabulary panoptic segmentation, meaning the ability to name and count arbitrary objects in a scene rather than a fixed label set, can be done directly on city-scale point clouds by a purely 3D model trained without any human-annotated data. The authors introduce a pseudo-labeling pipeline that renders synthetic images from the raw point cloud, projects vision-language features back onto the points, and clusters those features into pseudo-classes and pseudo-instances. They then train a superpoint graph transformer with mixture-of-experts blocks to predict CLIP-style feature vectors and instance labels in a single pass over the 3D data. The claim is that this is the first strictly 3D, non-projection-based open-vocabulary panoptic segmentation system applied to city-scale point clouds, demonstrated on the SensatUrban dataset.","feed_headline":"HAEC labels whole city scans in one 3D pass, no hand labels","feed_subtitle":"Open-vocabulary panoptic segmentation on city-scale point clouds, trained entirely on synthetic pseudo-labels from the raw scans.","key_machinery":"The load-bearing object is the superpoint graph transformer, a hierarchy of geometrically coherent point clusters connected as a graph, combined with a mixture-of-experts modification: at each level, top-2 expert graph-attention blocks are gated by each superpoint’s representation and the mean-aggregated relative positional encodings of its incoming edges. This hierarchy lets the model process millions of points as a small set of superpoints, so an entire city scene fits in one forward pass. The semantic head outputs a CLIP feature vector per superpoint instead of a class logit, and the training objective combines cosine reconstruction with a triplet loss that pulls same-pseudo-class superpoints together and pushes different pseudo-classes apart.","core_discovery":"The central discovery is that a strictly 3D superpoint-graph architecture can be trained with no human labels to produce open-vocabulary panoptic segmentation of city-scale point clouds. Training targets come from a synthetic labeling pipeline: cameras are cast into the point cloud, rendered images are screened by a lightweight vision-language model and given depth masks, per-pixel features from OpenSeg are projected back onto 3D points and averaged, and spherical k-means plus adaptive DBSCAN turn those features into pseudo-class and pseudo-instance labels. The backbone predicts CLIP feature vectors per superpoint rather than discrete classes, using cosine reconstruction and triplet losses. On SensatUrban the model reaches a semantic mIoU of 22.45 and on ScanNet a panoptic quality of 40.03, and it can answer long-tail text queries such as “the gate of King’s College” directly from the raw cloud.","pith_inferences":["The system’s ceiling is set by the vision-language feature extractor’s ability to read synthetic, point-cloud-style images, so improving that alignment, for example by fine-tuning a VLM on rendered-versus-real image pairs, should lift the whole pipeline more than any backbone change.","If pseudo-label noise can be reduced, the same superpoint mixture-of-experts backbone could be trained jointly on many unlabeled city scans, acting as an unsupervised foundation model for urban 3D understanding.","The stuff-versus-things split via an “object versus amorphous” text query is a testable shortcut: one could measure how sensitive final panoptic quality is to that binary threshold and replace it with a learned criterion.","The results suggest that expensive reconstruction-based open-vocabulary methods may be unnecessary for city-scale understanding once a trained 3D network can predict the same feature space directly."],"forward_implications":["Raw city-scale point clouds can be semantically queried and panoptically segmented in a single 3D pass, avoiding expensive per-scene 2D-to-3D projection at inference time.","Open-vocabulary 3D training no longer depends on hand-annotated 3D datasets, since synthetic images rendered from the cloud itself supply the supervision.","Long-tail and unique objects, such as specific buildings or colored cars, can be retrieved by free-form text queries even when they appear in no fixed class list.","The same pipeline transfers from indoor scenes to outdoor city data, suggesting a path toward foundation models trained on many unlabeled point clouds.","Because inference runs directly on the 3D data, preprocessing cost is paid once at training time rather than for every novel scene."],"supporting_citations":[{"why":"Supplies the OpenSeg per-pixel CLIP features that become the semantic training targets in the pseudo-label pipeline.","marker":"[7]"},{"why":"SuperCluster is the superpoint graph clustering panoptic framework that HAEC modifies with mixture-of-experts blocks and an open-vocabulary head.","marker":"[22]"},{"why":"SPT is the underlying hierarchical superpoint transformer architecture, providing graph self-attention and the coarse-to-fine encoder-decoder structure.","marker":"[21]"},{"why":"OpenScene is the 2D-to-3D distillation baseline whose feature-projection approach is adapted for pseudo-label generation.","marker":"[17]"},{"why":"CLIP defines the text-image embedding space that the predicted per-superpoint vectors are aligned to.","marker":"[19]"},{"why":"SensatUrban is the city-scale dataset on which the paper claims the first open-vocabulary segmentation result.","marker":"[11]"},{"why":"ScanNet is the dataset used to validate the instance pseudo-labels and compare panoptic quality.","marker":"[4]"},{"why":"MobileCLIP performs the image sanity check that filters invalid synthetic camera views before feature projection.","marker":"[25]"}],"fun_headline_variants":["Open-vocab 3D scene understanding scales to cities, no labels","HAEC: open-vocab city point cloud understanding without human labels","From raw city scans to open-vocab labels with zero manual work","HAEC unlocks open-vocab 3D understanding for entire cities","Label-free open-vocab segmentation on city-scale 3D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the OpenSeg feature vectors computed from synthetic, point-cloud-rendered images are semantically accurate enough to act as training labels; if that fails, no amount of backbone engineering can teach meaningful open-vocabulary semantics.","fun_headline_variants_meta":{"raw":{"variants":["Open-vocab 3D scene understanding scales to cities, no labels","HAEC: open-vocab city point cloud understanding without human labels","From raw city scans to open-vocab labels with zero manual work","HAEC unlocks open-vocab 3D understanding for entire cities","Label-free open-vocab segmentation on city-scale 3D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2529,"prompt_tokens":899,"completion_tokens":1630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":1534}},"tokens_in":515,"tokens_out":1630,"duration_ms":10985,"temperature":1.0,"reasoning_tokens":1534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:04:58.543689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Project the pseudo-label features on a held-out SensatUrban scene and test free-text queries such as “a red car” against manually verified regions; if retrieved points show no significant cosine-similarity advantage over random regions, or if the preprocessing oracle mIoU is within noise of the majority-class baseline, the pipeline cannot supply usable training targets.","supporting_citations":[{"cited_title":"Scal- ing open-vocabulary image segmentation with image-level labels","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenSeg per-pixel CLIP features that become the semantic training targets in the pseudo-label pipeline."},{"cited_title":"Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering","cited_arxiv_id":null,"evidence_quote":"SuperCluster is the superpoint graph clustering panoptic framework that HAEC modifies with mixture-of-experts blocks and an open-vocabulary head."},{"cited_title":"Efficient 3d semantic segmentation with superpoint transformer","cited_arxiv_id":null,"evidence_quote":"SPT is the underlying hierarchical superpoint transformer architecture, providing graph self-attention and the coarse-to-fine encoder-decoder structure."},{"cited_title":"Openscene: 3d scene understanding with open vocabular- ies","cited_arxiv_id":null,"evidence_quote":"OpenScene is the 2D-to-3D distillation baseline whose feature-projection approach is adapted for pseudo-label generation."},{"cited_title":"Towards semantic segmen- tation of urban-scale 3d point clouds: A dataset, benchmarks and challenges","cited_arxiv_id":null,"evidence_quote":"SensatUrban is the city-scale dataset on which the paper claims the first open-vocabulary segmentation result."},{"cited_title":"Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner","cited_arxiv_id":null,"evidence_quote":"ScanNet is the dataset used to validate the instance pseudo-labels and compare panoptic quality."},{"cited_title":"Mobile- clip: Fast image-text models through multi-modal reinforced training","cited_arxiv_id":null,"evidence_quote":"MobileCLIP performs the image sanity check that filters invalid synthetic camera views before feature projection."}],"review_version":1}