REVIEW 6 major objections 6 minor 1 cited by
OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model
T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that OpenMaskDINO3D, an LLM with a SEG token and object identifiers, sets new state-of-the-art results for 3D reasoning segmentation on ScanNet.
desk verdict 3D transplant of LISA-style reasoning segmentation with a clean architecture but a circular Mask3D pseudo-label evaluation that undermines the SOTA claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the <SEG> token, a special vocabulary entry whose output embedding becomes the object query for segmentation: after an MLP projection, the embedding seeds a transformer decoder that consumes supervoxel features as keys and values. Object identifiers (<OBJ> tokens) interleave the Uni3D 3D features, DINOv2 2D features, and positional encodings into a single sequence, letting the LLM reason over and reference specific instances. A superpoint pooling layer and hierarchical mask decoder handle large point clouds, and training combines an autoregressive language-model loss with BCE and DICE mask losses.
What would settle it
Take a random subset of the ScanNet reasoning-segmentation test samples, have human annotators draw the target-object masks, and recompute Acc@0.25, Acc@0.50, and mIoU for OpenMaskDINO3D against these human masks. If the scores fall well below the reported 54.21/39.14/39.81, the evaluation is measuring agreement with Mask3D proposals rather than ground truth.
Extended reading notes
Core claim
The discovery the paper argues for is that high-precision 3D segmentation masks can be generated directly from natural-language instructions by treating the LLM as the segmentation controller. Instead of training a separate open-vocabulary mask head, OpenMaskDINO3D inserts a learned <SEG> token into the LLM vocabulary; after the model reads a fused sequence of object identifiers, Uni3D point-cloud features, DINOv2 multi-view image features, and text, the last-layer embedding of <SEG> is projected through an MLP into the initial query of a transformer decoder, which reconstructs the mask over supervoxel features. Object identifiers <OBJ> give the LLM stable references to individual objects and are trained jointly with the model. The paper's central result is the ScanNet reasoning-segmentation table, where the model outperforms Reason3D and earlier open-vocabulary methods by wide margins, with ablation tables attributing the gain to the 2D image features and the object identifiers.
Load-bearing premise
The load-bearing premise is that the segmentation masks generated automatically by Mask3D can stand in for human-annotated ground truth when scoring reasoning segmentation.
Editorial extensions
If this is right
- A single LLM-based model can answer a reasoning-heavy 3D instruction and output the corresponding point-cloud mask, so referring segmentation, dense captioning, and 3D question answering share one architecture.
- The reported ScanNet scores imply that using the <SEG> embedding as the decoder query is a more effective way to ground open-vocabulary language in 3D than the prior pipeline of separate mask proposal and text matching.
- The zero-shot capability, with further gains from only 239 fine-tuning samples, suggests the approach can improve even with the scarce 3D scene-language data that limits the field.
- The new benchmark of over one thousand point-cloud-instruction-mask samples gives future work a common test bed for 3D reasoning segmentation.
Reading between the lines
- Because the reported ground truth is Mask3D-generated masks, the absolute scores likely reward agreement with that proposal distribution; a human-annotated test set would reveal how much of the gain reflects true open-world reasoning.
- The design implies that the language model compresses the entire scene description into a single <SEG> query vector; a testable extension is to emit multiple <SEG> tokens or decode several queries from one embedding to segment multiple objects in a single pass.
- Since the model depends on pre-trained object proposals to build the object-centric tokens, its recall is capped by that detector; testing on scenes with objects outside the proposal vocabulary would show whether the LLM can recover them.
- The same SEG-token mechanism could unify 2D and 3D reasoning segmentation in one multimodal LLM, since 2D LISA-style models use an analogous token.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes OpenMaskDINO3D, an LLM-based architecture for 3D reasoning segmentation that takes point clouds and text instructions as input and outputs instance masks. The model introduces a <SEG> token whose last-layer embedding is projected into a transformer decoder, together with learnable object identifiers <OBJ_i> that organize interleaved 3D (Uni3D) and 2D (DINOv2) features for the LLM. Training combines an autoregressive LLM loss with BCE and DICE mask losses, using data derived from the ScanRefer, Scan2Cap, ScanQA, and SQA3D benchmarks. The main empirical claim is state-of-the-art 3D reasoning segmentation on ScanNet, with Acc@0.25 of 54.21%, Acc@0.50 of 39.14%, and mIoU of 39.81% (Table 1), surpassing Reason3D by about 11, 7, and 8.6 percentage points, plus competitive results on grounding, dense captioning, and question answering (Table 2). However, the evaluation uses segmentation results generated by Mask3D as ground truth, which undermines the headline comparison as reported.
Significance. If the results held under a human-validated evaluation protocol, the work would be a useful step toward open-vocabulary 3D reasoning segmentation: the SEG-token embedding-as-mask design is a sensible transfer of 2D approaches to 3D, the object-identifier interleaving is clearly described, and the ablation tables (Tables 3 and 4) show consistent positive contributions from 2D features and object identifiers across segmentation metrics. The multi-task framing over four benchmarks is appropriate, and the paper is honest in stating that Mask3D outputs are adopted as ground truth. The paper does not contain machine-checked proofs, and although a code repository link is given, no data release, trained models, or detailed training configuration are provided, which limits verification. The significance of the state-of-the-art claim cannot currently be assessed because the evaluation protocol measures imitation fidelity to Mask3D proposals rather than reasoning segmentation accuracy against human annotations.
major comments (6)
- [Section 3, Table 1] The central claim of state-of-the-art performance rests on ground-truth masks produced by Mask3D: Section 3 states that 'we adopt the segmentation results generated by the Mask3D model from these 3D scene understanding training sets as the ground truth for comparison.' Since Section 4.3 trains the mask loss (BCE + DICE) against the same Mask3D outputs, the Acc@0.25, Acc@0.50, and mIoU figures in Table 1 measure how well the model reproduces Mask3D's proposal distribution, not how well it segments the object described by the instruction. Mask3D is a supervised class-agnostic instance segmenter whose proposals over- and under-segment targets, and any systematic bias in Mask3D becomes a learnable shortcut for a model trained and evaluated on the same labels. The 11-point gain over Reason3D is therefore unverifiable unless Reason3D was evaluated under the identical pseudo-label protocol, which the paper does not state. Because human instance annotations exist for ScanNet v2 and underlie the ScanRefer benchmark, the authors should re-evaluate on human-annotated ground-truth masks, or at minimum report agreement with Mask3D labels and a human-validated subsample separately.
- [Table 1, Section 5.3] The baselines that anchor the headline comparison are not described or cited: Reason3D appears in Table 1 with no reference anywhere in the paper, 3D-STMN is cited as [45] although the reference list contains only items [1]-[16], and Chat-Scene, discussed in Section 5.2 as state of the art on ScanRefer, is likewise uncited. Consequently it is impossible to determine the training data, evaluation ground truth, or metric details of the comparison, and in particular whether the baseline numbers are comparable to numbers computed against Mask3D-generated pseudo-masks. The authors should cite the original papers, state the evaluation protocol used for each baseline, and rerun the baselines under the same ground-truth regime used for OpenMaskDINO3D.
- [Tables 1, 3(b), and 4(b)] The reported numbers for the full model are internally inconsistent. Table 1 gives Acc@0.25 = 54.21%, Acc@0.50 = 39.14%, and mIoU = 39.81% for OpenMaskDINO3D on ScanNet, while Tables 3(b) and 4(b) report the configuration with both 2D features and object identifiers as 55.1%, 40.2%, and 40.3%. If these tables describe the same model and test set, the discrepancy (roughly one percentage point, and larger for mIoU) must be explained; if they describe different settings, such as a different number of fine-tuning samples or a different split of the reasoning-segmentation benchmark, that difference must be stated explicitly and the benchmark composition must be disclosed.
- [Section 3] The construction of the reasoning-segmentation benchmark is not described. The contributions claim 'over one thousand 3D point cloud-instruction-mask data samples' and 'minimal fine-tuning on 239 reasoning segmentation samples,' but Section 3 describes only the ScanRefer, Scan2Cap, ScanQA, and SQA3D task datasets, which do not natively contain reasoning-instruction-mask triplets. The paper never states how instructions were generated, how Mask3D proposals were matched to the objects referenced by each instruction, how many scenes and samples are in the training and test splits, or which of the reported numbers in Table 1 come from the zero-shot regime versus the 239-sample fine-tune. Without this information the main experimental claim is not reproducible, and the zero-shot claim made in the abstract and contributions is not operationalized.
- [Sections 4.1-4.3 and 5.1] Training and architecture details necessary for reproduction are absent. The overview in Section 4.1 says supervoxel features are simplified by a 'superpoint pooling layer,' while Section 4.2 says the scene is 'decomposed into object proposals using a pre-trained detector'; the relationship between these two descriptions is undefined, and the inherited 'spformer' backbone is not cited. No LoRA configuration (rank, alpha, dropout), number of object identifiers n, choice of LLM, optimizer, learning rate, batch size, number of epochs, or list of frozen parameters is given, and no variance across seeds is reported for any entry of Tables 1 or 2. These omissions make it impossible to judge whether the reported gains are robust or to reproduce the results from the text alone.
- [Section 5.1, Table 2] The evaluation pipeline for grounding and captioning is unspecified for a mask-output model. ScanRefer Acc@0.25/0.5 and Scan2Cap C@0.5/B-4@0.5 are defined with respect to IoU against ground-truth bounding boxes, but OpenMaskDINO3D outputs point-cloud masks; the paper does not state how masks are converted to boxes for these metrics, whether the conversion uses axis-aligned bounding boxes, or how failures of Mask3D proposal coverage affect the box IoU computation.
minor comments (6)
- [Table 3(a), Section 5.4] The paragraph on 2D features for question answering reports increases in accuracy, capture rate, and CIDEr, but it omits that SQA3D EM drops from 44.9 to 42.6 when 2D features are enabled; the drop should be reported in the same paragraph for balance.
- [References] The bibliography is incomplete: Table 1 cites items [31], [39], and [45], but only [1]-[16] are listed, and the text invokes Reason3D, Chat-Scene, SPFormer, LISA, and LEO without citations.
- [Section 4.3] The mask loss is written with an unspecified superscript '*' and the constraint '* in [seg]', while the text refers to M_seg; please unify the subscript and superscript notation.
- [Section 4.3] The total loss L = L_llm + L_mask has no weighting coefficients; please state the weights explicitly if they differ from 1.0.
- [Figure 1] The caption writes '[SEG]' whereas the text uses '<SEG>', and Figure 1 is difficult to read at the resolution provided; please unify the notation and improve the figure resolution.
- [Abstract] The abstract refers to 'large-scale ScanNet datasets,' but ScanNet is a single dataset; the benchmark claims in the contributions should be rephrased to name the constituent task datasets and their sample counts.
Circularity Check
Reported reasoning-segmentation SOTA is measured against Mask3D-generated pseudo-masks used as both training target and evaluation ground truth.
-
self definitional
[Section 3 (Dataset Collection), Section 4.3 (Training OpenMaskDINO3D), Section 5.1 (Evaluation Metrics)]
"Given the relatively poor performance of open methods in current open-vocabulary segmentation, we adopt the segmentation results generated by the Mask3D model from these 3D scene understanding training sets as the ground truth for comparison."
The benchmark's ground-truth mask M* is defined as Mask3D's prediction. Section 4.3 trains the mask head with L_mask = BCE(M*, M_hat) + DICE(M*, M_hat), and Section 5.1 reports mIoU and Acc@k as overlap between predicted and 'ground-truth 3D volumes.' Thus Table 1's Acc@0.25 54.21, Acc@0.50 39.14, and mIoU 39.81 measure agreement with Mask3D, not with human-annotated masks for the reasoning instruction. Because the model is trained on those same Mask3D outputs, high scores are partly forced by construction; the claimed 11-point advantage over Reason3D rests on a self-referential benchmark unless a human-validated test set is supplied.
full rationale
The central claim of state-of-the-art 3D reasoning segmentation in Table 1 is evaluated against Mask3D-generated pseudo-masks, which the paper explicitly adopts as ground truth. The same masks supervise the segmentation head via BCE and DICE losses. This makes the reported segmentation scores a measure of how well the model reproduces Mask3D's instance proposals, not of independent reasoning with human-verified masks. The comparison to Reason3D is therefore not controlled unless Reason3D uses the same pseudo-label protocol. The LLM and referring/captioning results are standard and not circular, so the score is 6 rather than higher; however, without a human-validated test set, the segmentation gain is partially self-referential.
Assumptions & free parameters
free parameters (3)
- Number of object identifiers (n) =
not reported
- LoRA configuration (rank, alpha, dropout) =
not reported
- Mask3D pseudo-label generation settings =
not reported
assumptions (5)
- domain assumption Uni3D provides reliable 3D point cloud features
- domain assumption DINOv2 features are useful for object-level understanding
- domain assumption SpFormer decoder is a suitable segmentation head
- ad hoc to paper Mask3D predictions can serve as ground truth for open-vocabulary segmentation evaluation
- domain assumption The LLM backbone and tokenizer are accessible and correctly integrated
invented entities (2)
-
<SEG> token
-
Object identifier tokens <OBJ_i>
Cite this review
Pith. "Pith review of OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model." pith.science (2026). https://pith.science/paper/M556G3KM
@misc{pith2026250604837,
author = {Pith},
title = {Pith review of: OpenMaskDINO3D : Reasoning 3D Segmentation via Large Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/M556G3KM}},
note = {Machine review of arXiv:2506.04837}
}
read the original abstract
Although perception systems have made remarkable advancements in recent years, particularly in 2D reasoning segmentation, these systems still rely on explicit human instruction or pre-defined categories to identify target objects before executing visual recognition tasks. Such systems have matured significantly, demonstrating the ability to reason and comprehend implicit user intentions in two-dimensional contexts, producing accurate segmentation masks based on complex and implicit query text. However, a comparable framework and structure for 3D reasoning segmentation remain absent. This paper introduces OpenMaskDINO3D, a LLM designed for comprehensive 3D understanding and segmentation. OpenMaskDINO3D processes point cloud data and text prompts to produce instance segmentation masks, excelling in many 3D tasks. By introducing a SEG token and object identifier, we achieve high-precision 3D segmentation mask generation, enabling the model to directly produce accurate point cloud segmentation results from natural language instructions. Experimental results on large-scale ScanNet datasets validate the effectiveness of our OpenMaskDINO3D across various tasks.
Figures
Forward citations
Cited by 1 Pith paper
-
MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs
MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.
Reference graph
Works this paper leans on
-
[1]
Henriques, Andrew Zisser- man, and Andrea Vedaldi
Yash Bhalgat, Iro Laina, João F. Henriques, Andrew Zisser- man, and Andrea Vedaldi. N2f2: Hierarchical scene under- standing with nested neural feature fields, 2024. 2
work page 2024
-
[16]
Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip
Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In ICCV, pages 2048–2059, 2023. 2
work page 2023
-
[2]
Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Col- laborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. NIPS, 36, 2023. 2
work page 2023
-
[3]
Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Image Token ScanRefer (Acc@0.5) Scan2Cap (C@0.5) ScanQA (CIDEr) SQA3D (EM) ✗ 27.8 18.4 23.3 44.9 ✓ 36.8 27.5 27.0 42.6 (a) 2D Features for Question Answering. Image Token Acc@0.25 Acc@0.50 mIoU ✗ 51.4 35.5 36.7 ✓ 55.1 40.2 40.3 (b) 2D Features for Segmentation. Table 3. Impact of 2D features on question-a...
work page 2023
-
[4]
Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation
Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. arXiv preprint,
-
[5]
Lerf: Language embedded radiance fields
Justin* Kerr, Chung Min* Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV, 2023. 2
work page 2023
-
[6]
Decomposing nerf for editing via feature field dis- tillation
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitz- mann. Decomposing nerf for editing via feature field dis- tillation. NIPS, 35:23311–23330, 2022. 2
work page 2022
-
[7]
Weakly supervised 3d open- vocabulary segmentation
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d open- vocabulary segmentation. NIPS, 36, 2024. 2
work page 2024
Show all 16 references
-
[8]
Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data
Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boular- ias, and Kostas Bekris. Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data. In Conference on Robot Learning, pages 1610–1620. PMLR, 2023. 2
2023
-
[9]
Openscene: 3d scene understanding with open vocabularies
Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, An- drea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 2
2023
-
[10]
Langsplat: 3d language gaussian splatting
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16084, 2023. 2
2023 arXiv
-
[11]
Learning transferable visual models from natural language supervi- sion
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...
2021
-
[12]
Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann
Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- Mask3D: Open-V ocabulary 3D Instance Segmentation. In NeurIPS, 2023. 2
2023
-
[13]
Vl-fields: Towards language-grounded neural implicit spatial representations
Nikolaos Tsagkas, Oisin Mac Aodha, and Chris Xiaoxuan Lu. Vl-fields: Towards language-grounded neural implicit spatial representations. arXiv preprint arXiv:2305.12427 ,
-
[14]
Open-fusion: Real-time open-vocabulary 3d map- ping and queryable scene representation
Kashu Yamazaki, Taisei Hanyu, Khoa V o, Thang Pham, Minh Tran, Gianfranco Doretto, Anh Nguyen, and Ngan Le. Open-fusion: Real-time open-vocabulary 3d map- ping and queryable scene representation. arXiv preprint arXiv:2310.03923, 2023. 2
-
[15]
Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding
Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962, 2023. 2
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.