Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAZER claims a training-free pipeline fuses frozen 2D vision-language masks into a real-time open-vocabulary 3D panoptic map, reporting state-of-the-art results on SceneNN, ScanNet200, and ScanNetv2 retrieval at 103.2 ms per frame.

desk verdict Useful online 3D mapping system with real speedups, but the semantic labeling protocol behind the headline mAP/mIoU numbers is never specified, so the evaluation has a load-bearing gap. read the letter →

arxiv 2505.15373 v1 pith:HP5E34PA submitted 2025-05-21 cs.CV cs.RO

Naman Patel , Prashanth Krishnamurthy , Farshad Khorrami This is my paper
classification cs.CVcs.RO
keywords open-vocabulary3Dsceneunderstandingzero-shotpanopticreconstructionRGB-Dsemanticmappinginstance-levelembeddingfusionR-treespatialindexingonlineinstancesegmentationnaturallanguageretrievalTSDFvolumetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a robot or AR system can build an open-vocabulary 3D map of a scene online, without any 3D training data, by taking per-frame instance masks and pixel features from a frozen 2D vision-language segmentation model and fusing them into a volumetric map as the camera moves. The proposed pipeline, RAZER, associates each new frame's oriented 3D bounding boxes with existing object tracks using R-tree spatial indexing and minimum-cost matching, and maintains up to three semantic embedding hypotheses per object so ambiguous or occluded views can be resolved later. If the reported numbers hold, the payoff is that arbitrary, previously unseen objects become queryable by natural language in real time (about 103 ms per frame), and the same map supports 3D instance segmentation, semantic segmentation, retrieval, and detection without retraining. The paper reports state-of-the-art or competitive results on SceneNN, ScanNet, ScanNetv2, ScanNet200, and Replica, including 79.15 mAP@50 on SceneNN instance segmentation and 24.7 mAP on ScanNet200.

What carries the argument

The load-bearing mechanism is the closed loop between an R-tree of oriented bounding boxes and an online embedding bank. Tracked objects are stored as AABB-enclosed OBBs in an R-tree, so each new detection queries only nearby candidates in expected O(log n) time; the final one-to-one association minimizes a weighted cost of 3D IoU dissimilarity and embedding distance using the Hungarian algorithm. Matched tracks update their OBB incrementally through scatter-matrix updates, so no point-cloud history is stored, and their semantic state merges the new embedding into one of up to three hypotheses when cosine similarity exceeds a threshold, otherwise spawning a new hypothesis. A per-voxel histogram records which instance label each voxel has most often, and objects whose voxel-support ratio falls below a threshold are pruned unless their semantic confidence is high, which is the mechanism intended to absorb inconsistent 2D masks while keeping identity through occlusions.

What would settle it

Take a real RGB-D sequence and corrupt the 2D instance masks by randomly splitting each mask into two fragments for some fraction of frames (or merging two objects into one mask), then measure SceneNN mAP@50; if the R-tree plus support-ratio pruning cannot rejoin or separate the affected tracks, the claimed robustness to 2D segmentation inconsistencies fails.

Watch

Extended reading notes

Core claim

RAZER's central claim is that open-vocabulary 3D panoptic mapping can be done in a fully zero-shot, online manner: a frozen 2D vision-language segmenter produces instance masks and dense semantic features (Eq. 2), which are lifted to 3D via depth back-projection, clustered with DBSCAN, and summarized as oriented bounding boxes. Each tracked object carries a semantic state of up to three concept embeddings updated by cosine-similarity voting (Eqs. 12-14), and an R-tree spatial index restricts a Hungarian bipartite match between new detections and existing tracks (Eqs. 7-8), so no global optimization or batch processing is required. Per-voxel label histograms and a support-ratio pruning rule (Eq. 15) maintain spatial consistency and remove spurious tracks while preserving objects under occlusion via a semantic-confidence override. The paper reports that this design surpasses prior methods on SceneNN (79.15 mAP@50), ScanNet200 (24.7 mAP), ScanNet semantics (0.393 mIoU) and ScanNetv2 retrieval (61.2% Top-1), at 103.2 ms per frame, and remains the leader under ORB-SLAM3-estimated poses (62.3 mAP@50).

Load-bearing premise

Everything rests on the frozen 2D vision-language segmentation model giving instance masks and pixel features that are consistent enough across frames for the 3D association to lock onto; the paper reports no experiment showing how performance degrades when those masks are fragmented, overlapping, or noisy.

Editorial extensions

If this is right

  • A robot can query the map in natural language while exploring and receive instance-level matches for objects never seen in training, because retrieval runs on the same fused embeddings that built the map.
  • The system stays online: at 103.2 ms per frame it is roughly 4x faster than the VolumePanoptic baseline (413.5 ms plus one-time map processing), so semantic mapping can keep up with live RGB-D streams.
  • Under noisy SLAM poses the method still leads the compared methods (62.3 mAP@50 on SceneNN with ORB-SLAM3, versus 58.82 for VolumePanoptic), indicating the geometric-semantic fusion tolerates pose error.
  • Because the 2D vision-language model is frozen and the 3D side is training-free, swapping in a better open-vocabulary segmenter should transfer to 3D without retraining the mapping pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported robustness to 2D segmentation inconsistencies is carried almost entirely by the R-tree/Hungarian association plus the support-ratio pruning heuristics; a stress test that artificially fragments or overlaps the input masks would reveal how much of the margin survives, since the paper does not measure this degradation directly.
  • The architecture is agnostic to how masks are produced, so the same association and embedding-fusion loop could be applied to outdoor or LiDAR-based mapping by rendering images for the 2D vision-language model, not just to indoor RGB-D streams.
  • The multi-hypothesis embedding bank opens an untapped path: when two hypotheses remain close in confidence, the system could deliberately re-observe the object to disambiguate, an active-perception behavior the paper does not exploit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes RAZER, a training-free framework for online zero-shot 3D panoptic mapping from posed RGB-D streams. It combines a frozen 2D vision-language segmentation model with TSDF-based volumetric reconstruction, instance-level OBB tracking via R-tree indexing and Hungarian matching, multi-hypothesis semantic embedding banks, and voxel-level label histograms. The authors evaluate on SceneNN, ScanNet200, ScanNet, ScanNetv2, and Replica, reporting state-of-the-art or competitive numbers for 3D instance segmentation, open-vocabulary instance segmentation, semantic segmentation, instance retrieval, and runtime, with a total runtime of 103.2 ms per frame on SceneNN.

Significance. If the claims are substantiated, the paper would make a useful contribution: it demonstrates that an online, training-free system can combine a frozen 2D VLM with geometric reconstruction to support open-vocabulary 3D instance retrieval and segmentation at interactive rates, while also reporting a substantial speed advantage over graph-based panoptic mapping baselines. The scope of evaluation across five datasets is a strength, as are the explicit runtime breakdowns and the use of standard benchmarks. However, the current manuscript does not fully specify several load-bearing components of the method and evaluation pipeline, in particular how class labels are produced for the semantic segmentation and open-vocabulary instance segmentation metrics, how the per-voxel histogram is updated, and what values are used for the many association and pruning thresholds. These omissions prevent the reader from verifying the central 'superior performance' claim and need to be addressed before the work can be accepted.

major comments (5)
  1. [Section IV-A, Eq. (1)] The voxel state vector in Eq. (1) includes a histogram h over observed instance labels and a maximum-likelihood label l, and Section IV-A states that these are 'also updated if the voxel lies within the truncated distance bound,' but no update rule for h is ever given. The support ratio r_i in Eq. (15) and the final instance segmentation in Section IV-H both depend on l, so this missing update rule leaves the core representation under-specified.
  2. [Section IV-H and Tables II/III] The system outputs per-voxel instance labels l in N and per-object embedding banks, but no protocol is described for converting these into the semantic class labels required by the ScanNet200 mAP and ScanNet/Replica mIoU metrics. If labels are assigned by thresholding cosine similarity to class-name prompts, the prompt set and the threshold sigma_sim used are unreported; if benchmark class names are used during evaluation, the zero-shot property of the numbers is ambiguous; if the metrics are computed class-agnostically, the comparisons to Open3DIS and OpenMask3D in Table II are invalid. This gap directly touches the headline 'superior performance' claim and must be resolved.
  3. [Sections IV-B, IV-C, IV-E, IV-F] Several free parameters are never given values or a selection procedure: DBSCAN epsilon and minimum points (Section IV-B), the Hungarian weights w_v and w_s in Eq. (8), the similarity threshold sigma_sim in Eq. (13), the support threshold tau_supp and confidence threshold tau_conf in Section IV-F, and the prune-frame count k. The robustness claims regarding inconsistent 2D masks rest on these parameters, so the experimental section needs to report concrete values and ideally a sensitivity study.
  4. [Table III] The semantic segmentation comparison is not controlled for the backbone. The table lists Ours with OVSeg only, while the text highlights the improvement over HOV-SG with ViT-H-14 (0.222 mIoU); the proper same-backbone comparison is HOV-SG with OVSeg (0.214 mIoU), and Ours with ViT-H-14 is missing. The current presentation overstates the margin by comparing to a different configuration.
  5. [Section V-D and Table IV] The retrieval evaluation is underspecified. The text says 'following the setting similar to [69]' but does not state which text prompts are used, whether the class names are part of the query set, or how the per-object embedding bank is converted into a predicted class for Top-1 classification. Because the headline retrieval number is 61.2% Top-1 accuracy, the exact inference-time labeling rule must be reported to assess whether this is a true zero-shot evaluation.
minor comments (5)
  1. [Section V-E] The text refers to 'Table V-E'; the table is numbered Table V, so the reference should be corrected.
  2. [Table IV] Several entries in the Table IV rows are garbled due to missing separators, e.g., the CLIP2Point w/ TP. row contains '11.572.292.4 86.1 3 34'; the table should be checked for formatting errors.
  3. [Eq. (8)] Equation (8) is typeset incorrectly: the minimization and the cost expression are not properly aligned, and it is unclear whether the displayed objective is a sum over all i,j of M_ij X_ij; please fix the equation and clarify that V_ij and S_ij are both dissimilarities/distance terms.
  4. [Section IV-B] The phrase 'must be tuned' for DBSCAN parameters acknowledges that the method has tunable components, but no guidance is given for choosing them other than balancing merging and splitting errors; reporting the actual values used in all experiments would resolve this ambiguity.
  5. [Section IV-C] The notation for the Hungarian cost uses M_ij for the cost matrix and also uses M^t for instance masks in Section IV-B; using a single symbol for both is confusing and should be disambiguated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; reported results rest on external benchmarks, with reproducibility gaps (unspecified label assignment and thresholds) rather than input-equivalent predictions.

full rationale

The paper is an empirical systems paper whose headline numbers (79.15 mAP@50 on SceneNN, 24.7 mAP on ScanNet200, 0.393 mIoU on ScanNet, 61.2% Top-1 retrieval on ScanNetv2) are all evaluated against external ground-truth annotations. The core pipeline (Eq. 2) uses a frozen 2D VLM to produce instance masks and embeddings, and Eqs. 12-14 average those embeddings over observations; the evaluation then compares the resulting fused embeddings or instance masks to benchmark labels. Nothing in the derivation chain defines the benchmark outcome in terms of the method's own outputs: ground-truth labels are not used to construct the map or to fit the reported parameters, and the relevant similarity computations (Eq. 13 and Section IV-I) are applied to text queries rather than to ground-truth labels. The few self-citations ([37], [40]) appear only in related-work context and are not load-bearing for the method's correctness. The main genuine concerns are reproducibility issues, not circularity: the paper does not specify how per-voxel semantic class labels are produced for the ScanNet/Replica mIoU and ScanNet200 mAP evaluations, and several thresholds (e.g., σ_sim, τ_supp, DBSCAN ε, w_v/w_s) are left unreported. Those omissions affect trust in the numbers but do not make the claimed results equivalent to the method's inputs by construction. Therefore no circular step is identified and the circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the quality of a frozen 2D VLM, on several hand-set thresholds not reported in the paper, and on standard geometric tools whose parameters are unspecified. These are not fitted to ground truth in the visible text, but they are free choices that a replication would need to fix.

free parameters (7)
  • DBSCAN radius epsilon = not reported
    Controls 3D clustering split and merge errors; Section IV-B.b says it must be tuned.
  • DBSCAN minimum points = not reported
    Same as epsilon, Section IV-B.b.
  • Similarity threshold sigma_sim = not reported
    Decides whether a new observation updates an existing embedding or starts a new hypothesis, Eq. 13, Section IV-E.
  • Support threshold tau_supp = not reported
    Prunes objects when voxel support ratio is low for k frames, Eq. 15, Section IV-F.
  • Confidence threshold tau_conf = not reported
    Overrides pruning for objects with high semantic confidence, Section IV-F.
  • Hungarian weights w_v and w_s = not reported
    Relative weight of geometric vs semantic cost in association, Eq. 8.
  • Unmatched prune frame count k = not reported
    Number of consecutive unmatched frames before a track is pruned, Section IV.C.c.
assumptions (5)
  • domain assumption The frozen pretrained VLM F produces instance masks and pixel embeddings that are reliable enough for 3D association.
    The pipeline lifts every mask to 3D and trusts mask boundaries; no mask-error model is given, Section IV.B.
  • domain assumption Camera poses are known or accurate enough for voxel fusion.
    TSDF integration and point lifting assume calibrated cameras and poses, either GT or ORB-SLAM3, Section III.
  • domain assumption A single rigid object corresponds to one 3D cluster after DBSCAN.
    Occlusions and overlapping masks are handled by DBSCAN, but split and merge behavior is parameter-dependent, Section IV.B.b.
  • ad hoc to paper The Hungarian cost with fixed weights combines geometry and semantics adequately.
    The linear combination of 1-IoU and L2 embedding distance in Eq. 8 is not learned or justified.
  • ad hoc to paper Three semantic hypotheses per object are sufficient.
    The embedding bank cap of 3 in Section IV-E is fixed without analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation." pith.science (2026). https://pith.science/paper/HP5E34PA

@misc{pith2026250515373,
  author       = {Pith},
  title        = {Pith review of: RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HP5E34PA}},
  note         = {Machine review of arXiv:2505.15373}
}
read the original abstract

Mapping and understanding complex 3D environments is fundamental to how autonomous systems perceive and interact with the physical world, requiring both precise geometric reconstruction and rich semantic comprehension. While existing 3D semantic mapping systems excel at reconstructing and identifying predefined object instances, they lack the flexibility to efficiently build semantic maps with open-vocabulary during online operation. Although recent vision-language models have enabled open-vocabulary object recognition in 2D images, they haven't yet bridged the gap to 3D spatial understanding. The critical challenge lies in developing a training-free unified system that can simultaneously construct accurate 3D maps while maintaining semantic consistency and supporting natural language interactions in real time. In this paper, we develop a zero-shot framework that seamlessly integrates GPU-accelerated geometric reconstruction with open-vocabulary vision-language models through online instance-level semantic embedding fusion, guided by hierarchical object association with spatial indexing. Our training-free system achieves superior performance through incremental processing and unified geometric-semantic updates, while robustly handling 2D segmentation inconsistencies. The proposed general-purpose 3D scene understanding framework can be used for various tasks including zero-shot 3D instance retrieval, segmentation, and object detection to reason about previously unseen objects and interpret natural language queries. The project page is available at https://razer-3d.github.io.

Figures

Figures reproduced from arXiv: 2505.15373 by the authors.

Figure 1
Figure 1. Pipeline overview of our proposed 3D scene understanding framework. Our system processes posed RGB-D inputs through open￾vocabulary segmentation for robust 3D instance tracking. Spatio-temporal feature aggregation fuses and prunes tracks while updating a panoptic map that enables online text-based 3D instance retrieval and segmentation tasks. 3D semantic-instance mapping addresses this limitation by simultaneously d… view at source ↗
Figure 2
Figure 2. System-level architecture of our RAZER framework. It processes RGB, depth, and pose inputs through three modules: (1) Instance Tracking to enable efficient feature updates, (2) Aggregation Manager to aggregate and fuse/prune instances and their corresponding coarse features, and (3) Map Update to update features at voxel level and their corresponding labels, thus generating a panoptic map that enables 3D scene under… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. QuASH: Using Natural-Language Heuristics to Query Visual-Language Robotic Maps

    cs.RO 2025-10 conditional novelty 6.0 of 10

    Querying VLM robot maps with an SVM trained on LLM-generated synonym/antonym embeddings outperforms cosine-threshold and single-antonym baselines on images and OpenSeg maps, but not consistently on LSeg maps.

Reference graph

Works this paper leans on

69 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [69]

    Openins3d: Snap and lookup for 3d open-vocabulary instance segmen- tation,

    Z. Huang, X. Wu, X. Chen, H. Zhao, L. Zhu, and J. Lasenby, “Openins3d: Snap and lookup for 3d open-vocabulary instance segmen- tation,” inProceedings of the European Conference on Computer Vision, 2024, pp. 169–185

    work page 2024

  2. [1]

    Associatively segmenting instances and semantics in point clouds,

    X. Wang, S. Liu, X. Shen, C. Shen, and J. Jia, “Associatively segmenting instances and semantics in point clouds,” inProceedings of the Confer- ence on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June 2019, pp. 4096–4105

    work page 2019

  3. [2]

    3d bird’s-eye- view instance segmentation,

    C. Elich, F. Engelmann, T. Kontogianni, and B. Leibe, “3d bird’s-eye- view instance segmentation,” inProceedings of the German Conference on Pattern Recognition, G. A. Fink, S. Frintrop, and X. Jiang, Eds., Dortmund, Germany, September 2019, pp. 48–61

    work page 2019

  4. [3]

    3d instance segmentation via multi-task metric learning,

    J. Lahoud, B. Ghanem, M. R. Oswald, and M. Pollefeys, “3d instance segmentation via multi-task metric learning,” inProceedings of the International Conference on Computer Vision, Seoul, South Korea, October 2019, pp. 9255–9265

    work page 2019

  5. [4]

    3d-sis: 3d semantic instance segmen- tation of RGB-D scans,

    J. Hou, A. Dai, and M. Nießner, “3d-sis: 3d semantic instance segmen- tation of RGB-D scans,” inProceedings of the Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June 2019, pp. 4421–4430

    work page 2019

  6. [5]

    Learning object bounding boxes for 3d instance segmenta- tion on point clouds,

    B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni, “Learning object bounding boxes for 3d instance segmenta- tion on point clouds,” inProceedings of the Advances in Neural Informa- tion Processing Systems, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´e-Buc, E. B. Fox, and R. Garnett, Eds., Vancouver, BC, Canada, December 20...

    work page 2019

  7. [6]

    Towards open vocabulary learning: A survey,

    J. Wu, X. Li, S. Xu, H. Yuan, H. Ding, Y . Yang, X. Li, J. Zhang, Y . Tong, X. Jiang, B. Ghanem, and D. Tao, “Towards open vocabulary learning: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 7, pp. 5092–5113, 2024

    work page 2024

  8. [7]

    A survey on open-vocabulary detection and segmentation: Past, present, and future,

    C. Zhu and L. Chen, “A survey on open-vocabulary detection and segmentation: Past, present, and future,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 8954–8975, 2024

    work page 2024

Show all 69 references
  1. [8]

    A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,

    M. Xu, Z. Zhang, F. Wei, Y . Lin, Y . Cao, H. Hu, and X. Bai, “A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,” inProceedings of the European Conference on Computer Vision, Tel Aviv, Israel, October 2022, pp. 736–753

    2022

  2. [9]

    Clio: Real-time task-driven open-set 3d scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3d scene graphs,”IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8921–8928, 2024

    2024

  3. [10]

    V olumetric Instance-Aware Semantic Mapping and 3D Object Discovery,

    M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto, “V olumetric Instance-Aware Semantic Mapping and 3D Object Discovery,”IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 3037–3044, July 2019

    2019

  4. [11]

    Panopticfusion: Online volumetric semantic mapping at the level of stuff and things,

    G. Narita, T. Seno, T. Ishikawa, and Y . Kaji, “Panopticfusion: Online volumetric semantic mapping at the level of stuff and things,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, SAR, China, November 2019, pp. 4205– 4212

    2019

  5. [12]

    Maskfusion: Real-time recog- nition, tracking and reconstruction of multiple moving objects,

    M. R ¨unz, M. Buffier, and L. Agapito, “Maskfusion: Real-time recog- nition, tracking and reconstruction of multiple moving objects,” in Proceedings of the International Symposium on Mixed and Augmented Reality, D. Chu, J. L. Gabbard, J. Grubert, and H. Regenbrecht, Eds., Muni...

    2018

  6. [13]

    Occuseg: Occupancy-aware 3d instance segmentation,

    L. Han, T. Zheng, L. Xu, and L. Fang, “Occuseg: Occupancy-aware 3d instance segmentation,” inProceedings of the Conference on Computer Vision and Pattern Recognition, Seattle, W A, USA, June 2020, pp. 2937– 2946

    2020

  7. [14]

    Hierarchical aggre- gation for 3d instance segmentation,

    S. Chen, J. Fang, Q. Zhang, W. Liu, and X. Wang, “Hierarchical aggre- gation for 3d instance segmentation,” inProceedings of the International Conference on Computer Vision, Montreal, QC, Canada, October 2021, pp. 15 447–15 456

    2021

  8. [15]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” inProceedings of the International Conference on Machine Learn...

    2021

  9. [16]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inProceedings of the International Conference on Machine Learning, Honolulu, HI, USA, July 2023, pp. 19 730–19 742

    2023

  10. [17]

    Pointclip: Point cloud understanding by CLIP,

    R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y . Qiao, P. Gao, and H. Li, “Pointclip: Point cloud understanding by CLIP,” inProceedings of the Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 2022, pp. 8542–8552

    2022

  11. [18]

    ULIP: learning a unified representation of language, images, and point clouds for 3d understanding,

    L. Xue, M. Gao, C. Xing, R. Mart ´ın-Mart´ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “ULIP: learning a unified representation of language, images, and point clouds for 3d understanding,” inProceed- ings of the Conference on Computer Vision and Pattern Recognit...

    2023

  12. [19]

    Con- ceptgraphs: Open-vocabulary 3d scene graphs for perception and plan- ning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappaet al., “Con- ceptgraphs: Open-vocabulary 3d scene graphs for perception and plan- ning,” inProceedings of the International Conference on Robotics and Automati...

    2024

  13. [20]

    Ins-conv: Incremental sparse convolution for online 3d segmentation,

    L. Liu, T. Zheng, Y . Lin, K. Ni, and L. Fang, “Ins-conv: Incremental sparse convolution for online 3d segmentation,” inProceedings of the Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 2022, pp. 18 953–18 962

    2022

  14. [21]

    PLA: language- driven open-vocabulary 3d scene understanding,

    R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, “PLA: language- driven open-vocabulary 3d scene understanding,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, June 2023, pp. 7010–7019

    2023

  15. [22]

    Context-aware alignment and mutual masking for 3d-language pre-training,

    Z. Jin, M. Hayat, Y . Yang, Y . Guo, and Y . Lei, “Context-aware alignment and mutual masking for 3d-language pre-training,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, June 2023, pp. 10 984–10 994

    2023

  16. [23]

    3d-vista: Pre- trained transformer for 3d vision and text alignment,

    Z. Zhu, X. Ma, Y . Chen, Z. Deng, S. Huang, and Q. Li, “3d-vista: Pre- trained transformer for 3d vision and text alignment,” inProceedings of the International Conference on Computer Vision, Paris, France, October 2023, pp. 2911–2921. 11

    2023

  17. [24]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning,

    S. Chen, H. Zhu, M. Li, X. Chen, P. Guo, Y . Lei, Y . Gang, T. Li, and T. Chen, “V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 11, pp. 7331–7347, 2024

    2024

  18. [25]

    Scan2cap: Context-aware dense captioning in RGB-D scans,

    D. Z. Chen, A. Gholami, M. Nießner, and A. X. Chang, “Scan2cap: Context-aware dense captioning in RGB-D scans,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, June 2021, pp. 3193–3203

    2021

  19. [26]

    Scanrefer: 3d object localization in RGB-D scans using natural language,

    D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object localization in RGB-D scans using natural language,” inProceedings of the European Conference on Computer Vision, Glasgow, UK, August 2020, pp. 202–221

    2020

  20. [27]

    Mask3d: Mask transformer for 3d semantic instance segmentation,

    J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe, “Mask3d: Mask transformer for 3d semantic instance segmentation,” in Proceedings of the International Conference on Robotics and Automa- tion, London, UK, May 2023, pp. 8216–8223

    2023

  21. [28]

    AGILE3D: attention guided interactive multi-object 3d segmentation,

    Y . Yue, S. Mahadevan, J. Schult, F. Engelmann, B. Leibe, K. Schindler, and T. Kontogianni, “AGILE3D: attention guided interactive multi-object 3d segmentation,” inProceedings of the International Conference on Learning Representations, Vienna, Austria, May 2024

    2024

  22. [29]

    Uni3d: Exploring unified 3d representation at scale,

    J. Zhou, J. Wang, B. Ma, Y .-S. Liu, T. Huang, and X. Wang, “Uni3d: Exploring unified 3d representation at scale,” inProceedings of the International Conference on Learning Representations, Vienna, Austria, May 2024

    2024

  23. [30]

    LERF: language embedded radiance fields,

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “LERF: language embedded radiance fields,” inProceedings of the International Conference on Computer Vision, Paris, France, October 2023, pp. 19 672–19 682

    2023

  24. [31]

    Open-nerf: Towards open vocabulary nerf decomposition,

    H. Zhang, F. Li, and N. Ahuja, “Open-nerf: Towards open vocabulary nerf decomposition,” inProceedings of the Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, January 2024, pp. 3444–3453

    2024

  25. [32]

    Gov-nesf: Generalizable open- vocabulary neural semantic fields,

    Y . Wang, H. Chen, and G. H. Lee, “Gov-nesf: Generalizable open- vocabulary neural semantic fields,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Seattle, W A, USA, June 2024, pp. 20 443–20 453

    2024

  26. [33]

    Semantically-aware neural radiance fields for visual scene understand- ing: A comprehensive review,

    T. Nguyen, A. Bourki, M. Macudzinski, A. Brunel, and M. Bennamoun, “Semantically-aware neural radiance fields for visual scene understand- ing: A comprehensive review,”CoRR, vol. abs/2402.11141, 2024

    2024 arXiv

  27. [34]

    Kimera: From SLAM to spatial perception with 3d dynamic scene graphs,

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: From SLAM to spatial perception with 3d dynamic scene graphs,”International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1510–1546, 2021

    2021

  28. [35]

    Mid-fusion: Octree-based object-level multi-instance dynamic SLAM,

    B. Xu, W. Li, D. Tzoumanikas, M. Bloesch, A. J. Davison, and S. Leutenegger, “Mid-fusion: Octree-based object-level multi-instance dynamic SLAM,” inProceedings of the International Conference on Robotics and Automation, Montreal, QC, Canada, May 2019, pp. 5231– 5237

    2019

  29. [36]

    Quadricslam: Dual quadrics from object detections as landmarks in object-oriented SLAM,

    L. Nicholson, M. Milford, and N. S ¨underhauf, “Quadricslam: Dual quadrics from object detections as landmarks in object-oriented SLAM,” IEEE Robotics and Automation Letters, vol. 4, no. 1, pp. 1–8, 2019

    2019

  30. [37]

    Semantic segmentation guided slam using vision and lidar,

    N. Patel, P. Krishnamurthy, and F. Khorrami, “Semantic segmentation guided slam using vision and lidar,” inProceedings of the International Symposium on Robotics, Munich, German, June 2018, pp. 1–7

    2018

  31. [38]

    SLAM++: simultaneous localisation and mapping at the level of objects,

    R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. J. Kelly, and A. J. Davison, “SLAM++: simultaneous localisation and mapping at the level of objects,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 2013, ...

    2013

  32. [39]

    vmap: Vectorised object mapping for neural field SLAM,

    X. Kong, S. Liu, M. Taher, and A. J. Davison, “vmap: Vectorised object mapping for neural field SLAM,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, June 2023, pp. 952–961

    2023

  33. [40]

    Tightly coupled semantic RGB-D inertial odometry for accurate long-term localization and mapping,

    N. Patel, F. Khorrami, P. Krishnamurthy, and A. Tzes, “Tightly coupled semantic RGB-D inertial odometry for accurate long-term localization and mapping,” inProceedings of the International Conference on Advanced Robotics, Belo Horizonte, Brazil, December 2019, pp. 523– 528

    2019

  34. [41]

    RO-MAP: real-time multi-object mapping with neural radiance fields,

    X. Han, H. Liu, Y . Ding, and L. Yang, “RO-MAP: real-time multi-object mapping with neural radiance fields,”IEEE Robotics and Automation Letters, vol. 8, no. 9, pp. 5950–5957, 2023

    2023

  35. [42]

    Fusion++: V olumetric object-level SLAM,

    J. McCormac, R. Clark, M. Bloesch, A. J. Davison, and S. Leutenegger, “Fusion++: V olumetric object-level SLAM,” inProceedings of the International Conference on 3D Vision, Verona, Italy, September 2018, pp. 32–41

    2018

  36. [43]

    Scenegraph- fusion: Incremental 3d scene graph prediction from rgb-d sequences,

    S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari, “Scenegraph- fusion: Incremental 3d scene graph prediction from rgb-d sequences,” Proceedings of the Conference on Computer Vision and Pattern Recog- nition, pp. 7515–7525, June 2022

    2022

  37. [44]

    Hi- erarchical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hi- erarchical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,” inProceedings of the Robotics: Science and Systems, Delft, Netherlands, July 2024

    2024

  38. [45]

    Llm-seg: Bridging image segmentation and large language model reasoning,

    J. Wang and L. Ke, “Llm-seg: Bridging image segmentation and large language model reasoning,” inProceedings of the International Conference on Computer Vision and Pattern Recognition, Seattle, W A, USA, June 2024, pp. 1765–1774

    2024

  39. [46]

    Llmformer: Large language model for open-vocabulary semantic segmentation,

    H. Shi, S. D. Dao, and J. Cai, “Llmformer: Large language model for open-vocabulary semantic segmentation,”International Journal of Computer Vision, August 2024

    2024

  40. [47]

    SQA3D: situated question answering in 3d scenes,

    X. Ma, S. Yong, Z. Zheng, Q. Li, Y . Liang, S. Zhu, and S. Huang, “SQA3D: situated question answering in 3d scenes,” inProceedings of the International Conference on Learning Representations, Kigali, Rwanda, May 2023

    2023

  41. [48]

    Scanqa: 3d question answering for spatial scene understanding,

    D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe, “Scanqa: 3d question answering for spatial scene understanding,” inProceedings of the Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 2022, pp. 19 107–19 117

    2022

  42. [49]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,

    P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas, “Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,” inProceedings of the European Conference on Computer Vision, Glasgow, UK, August 2020, pp. 422–440

    2020

  43. [50]

    Grounded 3d-llm with referent tokens,

    Y . Chen, S. Yang, H. Huang, T. Wang, R. Lyu, R. Xu, D. Lin, and J. Pang, “Grounded 3d-llm with referent tokens,”arXiv preprint arXiv:2405.10370, 2024

    2024 arXiv

  44. [51]

    3d-llm: Injecting the 3d world into large language models,

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan, “3d-llm: Injecting the 3d world into large language models,” inPro- ceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA, December 2023

    2023

  45. [52]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning,

    S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen, “Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning,” inProceedings of the Interna- tional Conference on Computer Vision and Pattern Recognition, Seattle, ...

    2024

  46. [53]

    LLM-Grounder: Open-vocabulary 3d visual grounding with large language model as an agent,

    J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai, “LLM-Grounder: Open-vocabulary 3d visual grounding with large language model as an agent,” inProceedings of the International Conference on Robotics and Automation, Yokohama, Japan, May 2024, pp. 7694–7701

    2024

  47. [54]

    Multi-object hallucination in vision-language models,

    X. Chen, Z. Ma, X. Zhang, S. Xu, S. Qian, J. Yang, D. F. Fouhey, and J. Chai, “Multi-object hallucination in vision-language models,” in Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, December 2024

    2024

  48. [55]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  49. [56]

    Open3DSG: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,

    S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski, “Open3DSG: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships,” inProceedings of the Conference on Computer Vision and Pattern Recognition, Seattle, W A, USA, J...

    2024

  50. [57]

    The hungarian method for the assignment problem,

    H. W. Kuhn, “The hungarian method for the assignment problem,”Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955

    1955

  51. [58]

    Scenenn: A scene meshes dataset with annotations,

    B. Hua, Q. Pham, D. T. Nguyen, M. Tran, L. Yu, and S. Yeung, “Scenenn: A scene meshes dataset with annotations,” inProceedings of the International Conference on 3D Vision, Stanford, CA, USA, October 2016, pp. 92–101

    2016

  52. [59]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” inProceedings of the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 2017, pp. 2432–2443

    2017

  53. [60]

    Scannet++: A high- fidelity dataset of 3d indoor scenes,

    C. Yeshwanth, Y . Liu, M. Nießner, and A. Dai, “Scannet++: A high- fidelity dataset of 3d indoor scenes,” inProceedings of the International Conference on Computer Vision, Paris, France, October 2023, pp. 12–22

    2023

  54. [61]

    Language-grounded indoor 3d semantic segmentation in the wild,

    D. Rozenberszki, O. Litany, and A. Dai, “Language-grounded indoor 3d semantic segmentation in the wild,” inProceedings of the European Conference on Computer Vision, S. Avidan, G. J. Brostow, M. Ciss ´e, G. M. Farinella, and T. Hassner, Eds., Tel Aviv, Israel, October 2022, pp...

    2022

  55. [62]

    The Replica dataset: A digital replica of indoor spaces,

    J. Straub, T. Whelan, L. Ma, Y . Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y . Yan, X. Pan, J. Yon, Y . Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasd...

    1906 arXiv

  56. [63]

    V olumetric instance-level seman- tic mapping via multi-view 2d-to-3d label diffusion,

    R. Mascaro, L. Teixeira, and M. Chli, “V olumetric instance-level seman- tic mapping via multi-view 2d-to-3d label diffusion,”IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3531–3538, 2022

    2022

  57. [64]

    Reconstructing interactive 3d scenes by panoptic mapping and CAD model alignments,

    M. Han, Z. Zhang, Z. Jiao, X. Xie, Y . Zhu, S. Zhu, and H. Liu, “Reconstructing interactive 3d scenes by panoptic mapping and CAD model alignments,” inProceedings of the International Conference on Robotics and Automation, Xi’an, China, May 2021, pp. 12 199–12 206

    2021

  58. [65]

    Multi-view fusion-based 3d object detection for robot indoor scene perception,

    L. Wang, R. Li, J. Sun, X. Liu, L. Zhao, H. S. Seah, C. K. Quah, and B. Tandianus, “Multi-view fusion-based 3d object detection for robot indoor scene perception,”Sensors, vol. 19, no. 19, p. 4092, 2019

    2019

  59. [66]

    Incremental instance-oriented 3d semantic mapping via rgb-d cameras for unknown indoor scene,

    W. Li, J. Gu, B. Chen, and J. Han, “Incremental instance-oriented 3d semantic mapping via rgb-d cameras for unknown indoor scene,” Discrete Dynamics in Nature and Society, vol. 2020, pp. 1–10, 2020

    2020

  60. [67]

    V olumetric semanti- cally consistent 3d panoptic mapping,

    Y . Miao, I. Armeni, M. Pollefeys, and D. Barath, “V olumetric semanti- cally consistent 3d panoptic mapping,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 924–12 931

    2024

  61. [68]

    Orb-slam3: An accurate open-source library for visual, visual– inertial, and multimap slam,

    C. Campos, R. Elvira, J. J. G. Rodr ´ıguez, J. M. Montiel, and J. D. Tard´os, “Orb-slam3: An accurate open-source library for visual, visual– inertial, and multimap slam,”IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.