Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that spatial reasoning in 3D point-cloud language models is a trainable two-stage process: first segment the objects relevant to a question, then feed their mask-pooled features back through the model's connector to…

desk verdict A clean two-stage reasoning-prior idea for 3D MLLMs with consistent gains, but the self-generated dataset and supervision make the headline results need release and human validation before they fully convince. read the letter →

arxiv 2506.23120 v1 pith:QPJUCV5L submitted 2025-06-29 cs.CV

Zhenhua Ning , Zhuotao Tian , Shaoshuai Shi , Guangming Lu , Daojing He , Wenjie Pei , Li Jiang This is my paper
classification cs.CV
keywords spatialreasoning3DpointcloudmultimodallargelanguagemodelsegmentationReasonSegvisualgroundingquestionansweringreferring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the reason 3D multimodal large language models fail on complex spatial instructions is not a lack of geometric information in point clouds but a lack of explicit reasoning structure. It proposes Relevant Reasoning Segmentation (R2S), a two-stage training procedure: first the model segments target-relevant objects as visual priors, then it feeds those objects' mask-pooled features back through the Q-Former so the language model can refine its answer and final mask. Alongside the method, the authors build 3D ReasonSeg, a 25,185-sample training set and 3,966-sample validation set of questions that require object functionality, visual attributes, and spatial relations to answer. They show that training with 3D ReasonSeg and R2S improves accuracy on ScanRefer, ScanQA, and 3D ReasonSeg itself, with the largest gains on the tasks that demand spatial reasoning.

What carries the argument

The central object is Relevant Reasoning Segmentation (R2S), a two-stage pipeline built on the [SEG]-token interface: a first instruction asks the model to segment question-related objects, giving masks $M^r$; mask-pooling aggregates super-point features $f^p$ into per-object features $f^r = f^p \times M^r$; a second instruction embeds these features into the Q-Former along with the question ($w' \oplus f^r$), refining the latent queries to $q'_l$ and the object features to $f'_r$; the LLM then generates the answer and the mask decoder produces the final mask from the refined [SEG] hidden state. The mechanism that carries the argument is that the first stage provides a concrete visual prior that the second stage can attend to, rather than requiring the model to infer relevant objects and target simultaneously.

What would settle it

Take the 3D ReasonSeg validation set, have humans re-annotate every question's target and relevant objects, and recompute the reported gIoU and Acc@50 on the cleaned subset; if R2S's gains shrink to noise, the measured improvement is an artifact of noisy LLM-generated labels.

Watch

Extended reading notes

Core claim

The central claim is that spatial reasoning in 3D point-cloud language models can be trained by emulating human two-step search: first recognize the objects that are relevant to the question, then inspect those objects closely to locate the target. Compared with direct instruction tuning, this two-step decomposition improves Acc@50 on ScanRefer from 38.7 to 48.7 and gIoU on 3D ReasonSeg from 29.2 to 33.1, with consistent gains on ScanQA's BLEU-4, CIDEr, METEOR, and Rouge-L. The authors argue the mechanism is the visual prior: mask-pooling the segmented relevant objects produces instance-level features that carry spatial detail ordinary text names cannot, and feeding these features through the Q-Former in a second pass lets the model refine its latent queries before the final mask is decoded. They also claim existing datasets are too simple (e.g., 'in front of' is ambiguous in ScanRefer) and that 3D ReasonSeg supplies the missing complex spatial reasoning supervision.

Load-bearing premise

The load-bearing premise is that the large-language-model-generated object lists and question-answer pairs, after rule-based filtering, are accurate enough to serve as ground truth for both training supervision and the 3D ReasonSeg evaluation; the authors themselves note that noise remains after cleaning.

Editorial extensions

If this is right

  • R2S can be layered onto any 3D MLLM that already outputs [SEG] tokens, so the two-stage recipe is a drop-in training strategy rather than a new architecture.
  • 3D ReasonSeg provides a reusable benchmark whose questions deliberately require object functionality, visual attributes, and spatial relations; models trained on it generalize better to ScanRefer and ScanQA.
  • The relevant-object augmentation (randomly dropping or adding priors during training) makes the model robust to imperfect first-stage predictions at inference.
  • The largest improvements appear exactly on spatial-reasoning metrics (Acc@50 and gIoU), supporting the claim that explicit prior learning addresses the identified failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to iterate the two stages more than once, using the refined mask as the prior for another pass; the paper's architecture would support it without new components.
  • Since the priors are mask-level features rather than text, the method implicitly teaches the model to ground relational language in geometry; that suggests it could transfer to embodied instruction following where object positions matter.
  • Because both the dataset and the target-relevant annotations are LLM-generated, a human-annotated subset of 3D ReasonSeg would be the cleanest way to separate genuine reasoning gains from alignment with the data generator's annotation style.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Relevant Reasoning Segmentation (R2S), a two-stage framework for 3D point-cloud segmentation guided by spatial reasoning. In Stage 1, the model predicts masks for objects relevant to a question, and these masks are used to pool visual features that are injected as reasoning priors into a revised instruction. In Stage 2, the Q-Former and LLM are run again with the prior-augmented instruction to produce the final segmentation. The paper also introduces 3D ReasonSeg, a dataset of 25,185 training and 3,966 validation samples generated by Llama 3.1 from SceneVerse object annotations and filtered with rule-based heuristics. Experiments on ScanRefer, ScanQA, and 3D ReasonSeg compare the baseline, baseline trained with 3D ReasonSeg, and the full R2S method, reporting consistent improvements across all metrics, with ablations on the prior-guided refinement, the number of latent queries, and the text-loss weight.

Significance. If the results are robust, R2S provides a conceptually simple and effective way to inject explicit reasoning priors into 3D multimodal LLMs, and 3D ReasonSeg could become a useful resource for evaluating spatial reasoning. The paper's strengths include a clear problem formulation, a well-motivated two-stage design, and extensive ablations. The improvements on ScanRefer Acc@50 (38.7 to 48.7) and 3D ReasonSeg gIoU (29.2 to 33.1) are substantial. However, the benchmark and the prior supervision are both generated by the same LLM, and the paper does not demonstrate that the generated labels are accurate enough to serve as reliable ground truth. The lack of independent validation of the dataset, combined with several internal specification gaps, makes the central claim of robust spatial-reasoning gains provisional pending additional evidence.

major comments (4)
  1. [Sec. 3.3, Appendix A, and Limitation] The claim that 3D ReasonSeg has 'precise annotations' and is maintained through 'manual verification processes' is not supported by the described procedure. Appendix A states that manual examination was used to formulate filtering strategies, but no spot-check rate, inter-annotator agreement, or correction counts are reported. The Limitation section concedes that 'some noise remains in our dataset even after data cleaning procedures.' Because the same Llama 3.1-generated labels supervise the R2S prior stage (Sec. 3.2, Eqs. 6-7) and also define the 3D ReasonSeg evaluation targets, the gIoU gains in Table 2 (29.2 to 33.1) and the R2S-specific improvements in Table 3 may partly reflect the model's ability to imitate the generator's biases rather than to perform human-validated spatial reasoning. Please provide an independent human-annotated subset with per-sample accuracy statistics, or evaluate on an existing human-annotated reasoning benchmark, to demonstrate that the gains transfer beyond the self-generated benchmark.
  2. [Sec. 3.2, Eqs. (6)-(9)] The handling of multiple relevant-object masks is underspecified. Eq. (7) defines f_r as a single feature tensor via f_p × M_r, implying one mask, but the text and Fig. 4 describe n relevant objects with features f^1_r, ..., f^n_r. The paper does not state whether M_r aggregates masks by summation or averaging, whether each object is pooled separately, how many feature vectors are produced, or how these vectors are inserted into the instruction embedding w′ ⊕ f_r (e.g., as additional tokens or as a sequence concatenated to the Q-Former query). Without this detail, the architecture is not reproducible. Please specify the exact tensor operations, the number and shape of the prior features, and the exact injection mechanism for scenes with multiple relevant objects.
  3. [Sec. 4.2, Table 5] The prose states that 'setting λtxt to 1.0 achieves optimal balance in performance,' but Table 5 shows that λtxt=0.5 yields the best or tied-best score on all metrics except ScanRefer Acc@25 and Acc@50, where λtxt=0.1 is slightly higher. For example, ScanQA CIDEr is 75.2 for λtxt=0.5 versus 72.6 for λtxt=1.0, and 3D ReasonSeg gIoU is 29.2 versus 28.6. This is a direct contradiction between the text and the table. Please correct the text to reflect the actual optimum and state how sensitive the main conclusions are to this hyperparameter.
  4. [Sec. 3.3 and Table 2] The paper does not disclose whether the scenes used in 3D ReasonSeg overlap with the scenes in the ScanQA and ScanRefer evaluation sets. If 3D ReasonSeg is built from ScanNet or SceneVerse scenes that also appear in ScanQA/ScanRefer, then training on 3D ReasonSeg could inflate the cross-benchmark gains (e.g., ScanRefer Acc@50 from 38.7 to 48.7) through scene-level memorization rather than transferable spatial reasoning. Please report the scene-level split, the number of unique scene IDs in each split, and verify that the evaluation scenes are disjoint from all training scenes.
minor comments (6)
  1. [Abstract and Sec. 1] The phrase 'manual verification processes' overstates the procedure described in Appendix A, which reports only manual examination to design filtering strategies. Consider replacing it with 'rule-based filtering informed by manual inspection.'
  2. [Sec. 1, Contributions] The first contribution bullet says 'Relevant Segmentation (R2S)' but the method is introduced as 'Relevant Reasoning Segmentation (R2S).' Please fix the typo.
  3. [Table 3] The caption says 'Ablation study on the Prior-guided Refinement,' but the meaning of the 'wo PR' and 'text-based' rows is not fully explicit. Please state in the caption whether all rows include training on 3D ReasonSeg.
  4. [Tables 2-5] No variance or significance tests are reported. Providing repeated-run mean and standard deviation for at least the main reported numbers would help assess the stability of the improvements, especially for metrics where the differences are a few points.
  5. [Sec. 4.1 and Eq. (11)] The sentence 'we assign equal weights of 1 to the BCE and DICE loss' is redundant with the unweighted sum in Eq. (11); clarify whether any weights are applied.
  6. [Eq. (7)] The multiplication symbol in f_r = f_p × M_r is ambiguous. Please specify whether this is elementwise multiplication between the super-point features and the mask scores, and state the dimensions of the result.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity: the framework and benchmark are self-contained empirical contributions, with external benchmarks providing independent support.

full rationale

The paper's central claims are empirical: R2S improves segmentation and QA metrics, and the new 3D ReasonSeg dataset provides training and evaluation samples. The method equations (Eqs. 1-11) describe a feed-forward architecture with a two-stage prior and refinement; no equation-level reduction to the input labels or to a fitted parameter occurs. The ScanRefer and ScanQA results are evaluated against human-created benchmarks, and the R2S gains on those benchmarks are not derived from the paper's own dataset. The 3D ReasonSeg benchmark is generated with Llama 3.1 from SceneVerse annotations, and Section 3.4 further uses Llama 3.1 to produce target-relevant object supervision for R2S; the Appendix A filtering is rule-based and the Limitation section concedes 'some noise remains in our dataset even after data cleaning procedures.' This is a legitimate benchmark-validity concern — the test distribution may share systematic LLM bias with the training supervision — but it is not circularity in the derivation sense: the benchmark labels and the model predictions are not the same quantity by construction, no fitted parameter is renamed as a prediction, and the external benchmarks provide an independent check. The few self-citations (e.g., following LISA [26] for the [SEG] token and gIoU metric) are standard methodological borrowing and are not load-bearing for the central claim. Therefore the paper merits a circularity score of 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of LLM-generated annotations and a specific architecture; the paper contributes an empirical method and dataset rather than a formal derivation, and reproducibility artifacts are absent.

free parameters (5)
  • Text loss weight lambda_txt = 0.5 (Table 5)
    Chosen by ablation; the paper text incorrectly says 1.0 is optimal. Affects balance of text and segmentation losses in Eq. (11).
  • Number of Q-Former latent queries = 32
    Selected as best in Table 4; controls how much point cloud information is compressed before the LLM.
  • Learning rate schedule = 1e-4 to 1e-6 cosine
    AdamW hyperparameter; standard, but hand-selected for training in Sec. 4.1.
  • Distance threshold for 'near' filtering = not reported
    Appendix A filters 'near' relations using an unreported distance threshold; this changes dataset composition and difficulty.
  • Relevant-object augmentation omission/addition probability = not reported
    Sec. 3.4 introduces random omission/addition of relevant objects, but the probability or schedule is not specified; it affects train-inference discrepancy handling.
assumptions (5)
  • domain assumption OneFormer3D encoder and OPT 1.3B frozen LLM provide sufficient capacity for spatial reasoning after fine-tuning other modules.
    Used as backbone in Sec. 3.1 and 4.1; results depend on this architecture choice.
  • ad hoc to paper Llama 3.1-generated question-reasoning-answer triples and relevant-object labels are correct enough to serve as ground truth after rule-based filtering.
    Dataset construction in Sec. 3.3 and Appendix A relies on LLM generation; Limitation section admits residual noise.
  • domain assumption Super-point aggregation preserves sufficient spatial detail for precise segmentation.
    Sec. 3.1 uses super-points to reduce compute; if aggregation loses spatial cues, the central claim about spatial reasoning weakens.
  • domain assumption Per-scene gIoU is an appropriate metric for evaluating reasoning-based segmentation.
    Metric from [26] used in Sec. 4.1; it treats every scene equally and may not reflect object-level accuracy.
  • ad hoc to paper Mask-pooling in Eq. (7) yields features that preserve target-relevant object identity and position.
    This is the mechanism linking Step 1 segmentation to Step 2 refinement; no theoretical guarantee is given that f_r retains spatial details.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation." pith.science (2026). https://pith.science/paper/QPJUCV5L

@misc{pith2026250623120,
  author       = {Pith},
  title        = {Pith review of: Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPJUCV5L}},
  note         = {Machine review of arXiv:2506.23120}
}
abstract

Recent advances in point cloud perception have demonstrated remarkable progress in scene understanding through vision-language alignment leveraging large language models (LLMs). However, existing methods may still encounter challenges in handling complex instructions that require accurate spatial reasoning, even if the 3D point cloud data provides detailed spatial cues such as size and position for identifying the targets. To tackle this issue, we propose Relevant Reasoning Segmentation (R$^2$S), a reasoning-based segmentation framework. The framework emulates human cognitive processes by decomposing spatial reasoning into two sequential stages: first identifying relevant elements, then processing instructions guided by their associated visual priors. Furthermore, acknowledging the inadequacy of existing datasets in complex reasoning tasks, we introduce 3D ReasonSeg, a reasoning-based segmentation dataset comprising 25,185 training samples and 3,966 validation samples with precise annotations. Both quantitative and qualitative experiments demonstrate that the R$^2$S and 3D ReasonSeg effectively endow 3D point cloud perception with stronger spatial reasoning capabilities, and we hope that they can serve as a new baseline and benchmark for future work.

Figures

Figures reproduced from arXiv: 2506.23120 by the authors.

Figure 1
Figure 1. In contrast to the baseline model in Sec. 3.1, we introduce Relevant Reasoning Segmentation to explicitly guide the model’s [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The image on the left depicts the LLM’s response to the question “ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The baseline model consists of a visual backbone for point feature encoding, a Q-Former that connects visual-language represen [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The proposed Relevant Reasoning Segmentation framework. Stage 1 segments question-related objects as priors ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: This illustrates a data sample from ScanRefer, where the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Examples of target-relevant data. The first row shows an [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Visual qualitative results on ScanQA and 3D ReasonSeg. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The prompt for yielding 3D ReasonSeg. Generate step-by-step reasoning for question-answer pairs based on 3D scene object annotations. Each question-answer pair follows this structure: {question_id: <question_id>, question: <question>, answer: <answer>}. Object annotati…
Figure 10
Figure 10. Figure 10: The prompt for generating target-relevant objects for ScanQA and ScanRefer. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: More template for Relevant Reasoning Segmentation. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: More qualitative results on 3D ReasonSeg. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: More qualitative results on scene description and embodied planning. [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Fine-tuning VLMs on balanced synthetic images with known object positions improves absolute-position accuracy on real COCO images by 13-21 points while using two orders of magnitude less data.

Reference graph

Works this paper leans on

77 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129– 19139, 2022. 2, 6, 7, 12

    work page 2022

  3. [3]

    Semantic scene com- pletion via integrating instances and scene in-the-loop

    Yingjie Cai, Xuesong Chen, Chao Zhang, Kwan-Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene com- pletion via integrating instances and scene in-the-loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324–333, 2021. 2

    work page 2021

  4. [4]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020. 2, 6, 7, 12

    work page 2020

  5. [5]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522–20535,

    work page

  6. [6]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428–26438, 2024. 1, 2, 3, 6, 7

    work page 2024

  7. [7]

    Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal ob- ject detection

    Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Che- ung, Hang Xu, and Hongsheng Li. Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal ob- ject detection. In European Conference on Computer Vision, pages 680–697. Springer, 2022. 2

    work page 2022

  8. [8]

    Trajectoryformer: 3d object tracking transformer with predictive trajectory hypotheses

    Xuesong Chen, Shaoshuai Shi, Chao Zhang, Benjin Zhu, Qiang Wang, Ka Chun Cheung, Simon See, and Hong- sheng Li. Trajectoryformer: 3d object tracking transformer with predictive trajectory hypotheses. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18527–18536, 2023. 2

    work page 2023

Show all 77 references
  1. [9]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370, 2024. 2, 7

    2024 arXiv

  2. [10]

    Reslt: Residual learning for long-tailed recog- nition

    Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recog- nition. IEEE transactions on pattern analysis and machine intelligence, 45(3):3695–3706, 2022. 12

    2022

  3. [11]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 6

    2017

  4. [12]

    Mta- clip: Language-guided semantic segmentation with mask- text alignment

    Anurag Das, Xinting Hu, Li Jiang, and Bernt Schiele. Mta- clip: Language-guided semantic segmentation with mask- text alignment. arXiv preprint arXiv:2407.21654, 2024. 2

    2024 arXiv

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 12, 14

    2018 arXiv

  6. [14]

    Pla: Language-driven open- vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open- vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7010–7019, 2023. 2

    2023

  7. [15]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et.al. The llama 3 herd of models, 2024. 5, 6, 12

    2024

  8. [16]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understand- ing, generation, and instruction following

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xi- anzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xi- anzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understand- ing, generation, and instruction following. arXiv prepri...

    2023 arXiv

  9. [17]

    Imagebind-llm: Multi-modality instruction tun- ing

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tun- ing. arXiv preprint arXiv:2309.03905, 2023. 2

    2023 arXiv

  10. [18]

    3d-llm: Inject- ing the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inject- ing the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494,

  11. [19]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168, 2023. 1, 2, 3, 7

    2023 arXiv

  12. [20]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023. 2

    2023 arXiv

  13. [21]

    Multi- view transformer for 3d visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi- view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524–15533, 2022. 2, 7

    2022

  14. [22]

    Sceneverse: Scaling 3d vision-language learn- ing for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learn- ing for grounded scene understanding. arXiv preprint arXiv:2401.09340, 2024. 5, 6

    2024 arXiv

  15. [23]

    Guided point contrastive learn- ing for semi-supervised point cloud semantic segmentation

    Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Guided point contrastive learn- ing for semi-supervised point cloud semantic segmentation. 9 In Proceedings of the IEEE/CVF international conference on computer vision, pages 6423–6432, 2021. 12

    2021

  16. [24]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 3

    2023 arXiv

  17. [25]

    Oneformer3d: One transformer for unified point cloud segmentation

    Maxim Kolodiazhnyi, Anna V orontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20943–20953, 2024. 2, 6, 12, 14

    2024

  18. [26]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 3, 7

    2024

  19. [27]

    Step-dpo: Step-wise preference op- timization for long-chain reasoning of llms

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xian- gru Peng, and Jiaya Jia. Step-dpo: Step-wise preference op- timization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024. 12

    2024 arXiv

  20. [28]

    Large-scale point cloud semantic segmentation with superpoint graphs

    Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4558–4567, 2018. 3

    2018

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730– 19742. PMLR, 2023. 1, 3, 12, 14

    2023

  22. [30]

    Uni3d-llm: Unifying point cloud perception, generation and editing with large language models

    Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arXiv preprint arXiv:2402.03327, 2024. 2

    2024 arXiv

  23. [31]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  24. [32]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2

    2022 arXiv

  25. [33]

    An end-to-end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2906–2917, 2021. 2

    2021

  26. [34]

    Boosting few-shot 3d point cloud segmentation via query-guided enhancement

    Zhenhua Ning, Zhuotao Tian, Guangming Lu, and Wenjie Pei. Boosting few-shot 3d point cloud segmentation via query-guided enhancement. In Proceedings of the 31st ACM international conference on multimedia, pages 1895–1904,

    1904

  27. [35]

    Hierarchi- cal dense correlation distillation for few-shot segmentation

    Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchi- cal dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23641–23651, 2023. 12

    2023

  28. [36]

    Scalable language model with generalized contin- ual learning

    Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, and Jiaya Jia. Scalable language model with generalized contin- ual learning. arXiv preprint arXiv:2404.07470, 2024. 12

    2024 arXiv

  29. [37]

    Oa-cnns: Omni- adaptive sparse cnns for 3d semantic segmentation

    Bohao Peng, Xiaoyang Wu, Li Jiang, Yukang Chen, Heng- shuang Zhao, Zhuotao Tian, and Jiaya Jia. Oa-cnns: Omni- adaptive sparse cnns for 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21305–21315, 2024. 12

    2024

  30. [38]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815–824, 2023. 2

    2023

  31. [39]

    Explore the potential of clip for training-free open vocab- ulary semantic segmentation

    Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training-free open vocab- ulary semantic segmentation. In European Conference on Computer Vision, pages 139–156. Springer, 2024. 12

    2024

  32. [40]

    Pointr- cnn: 3d object proposal generation and detection from point cloud

    Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointr- cnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–779,

  33. [41]

    Open- mask3d: Open-vocabulary 3d instance segmentation

    Ayc ¸a Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- mask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.13631, 2023. 2

    2023 arXiv

  34. [42]

    Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models

    Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Han- tao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European Conference on Computer Vision, page...

    2024

  35. [43]

    Learning shape-aware embedding for scene text detection

    Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. Learning shape-aware embedding for scene text detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4234–4243, 2019. 12

    2019

  36. [44]

    Prior guided feature enrich- ment network for few-shot segmentation

    Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrich- ment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050– 1065, 2020. 12

    2020

  37. [45]

    Adaptive perspective distillation for semantic segmen- tation

    Zhuotao Tian, Pengguang Chen, Xin Lai, Li Jiang, Shu Liu, Hengshuang Zhao, Bei Yu, Ming-Chang Yang, and Jiaya Jia. Adaptive perspective distillation for semantic segmen- tation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1372–1387, 2022. 12

    2022

  38. [46]

    Generalized few-shot semantic segmentation

    Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. Generalized few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11563–11572, 2022. 12

    2022

  39. [47]

    Learning context-aware classifier for semantic segmentation

    Zhuotao Tian, Jiequan Cui, Li Jiang, Xiaojuan Qi, Xin Lai, Yixin Chen, Shu Liu, and Jiaya Jia. Learning context-aware classifier for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2438– 2446, 2023. 12 10

    2023

  40. [48]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. 1, 2, 7

    2023 arXiv

  41. [49]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 1

    2023 arXiv

  42. [50]

    Pointllm: Empowering large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911, 2023. 1, 2

    2023 arXiv

  43. [51]

    Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7...

    2024

  44. [52]

    Lisa++: An improved baseline for reasoning segmentation with large language model.arXiv preprint arXiv:2312.17240, 2023

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model.arXiv preprint arXiv:2312.17240, 2023. 12

    2023 arXiv

  45. [53]

    Unified language-driven zero-shot domain adaptation

    Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23407–23415, 2024

    2024

  46. [54]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19792–19802, 2025. 12

    2025

  47. [55]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. 1

    2023 arXiv

  48. [56]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 1

    2023 arXiv

  49. [57]

    Opt: Open pre-trained trans- former language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068,

    arXiv

  50. [58]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225–15236, 2023. 2, 7

    2023

  51. [59]

    3dvg- transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937, 2021. 7

    2021

  52. [60]

    Empowering 3d visual grounding with reasoning capabilities

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Empowering 3d visual grounding with reasoning capabilities. arXiv preprint arXiv:2407.01525, 2024. 2, 7

    2024 arXiv

  53. [61]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1

    2023 arXiv

  54. [62]

    in the right of

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 7 11 A. Data construction details...

    2023

  55. [63]

    Focus on detecting and distinguishing objects

    Generate challenging object-centric questions using the annotations: a. Focus on detecting and distinguishing objects. The answer should specify one or more objects. b. Include at least two of the objects' uses, appearance, positions, and sizes. c. Require logical reasoning to...

  56. [64]

    Provide step-by-step reasoning for each question follow these steps:

  57. [65]

    Analyze the properties and spatial relationship of objects in the scene

  58. [66]

    Analyze the question and identify question-related objects

  59. [67]

    Construct a logical step-by-step reasoning based on your analyzation to answer the question

  60. [68]

    Present the reasoning steps clearly

  61. [69]

    For multiple objects of the same category, use [<ins_id>][<ins_id>][<ins_id>], etc

    Use [<ins_id>] to refer objects in reasoning steps . For multiple objects of the same category, use [<ins_id>][<ins_id>][<ins_id>], etc

  62. [70]

    Answer the question accurately

  63. [71]

    Only use objects from the annotations

  64. [72]

    Don't output anything else

    Output format: 'question;reasoning;answer'. Don't output anything else. Now generate for following annotations: Figure 9. The prompt for yielding 3D ReasonSeg. Generate step-by-step reasoning for question-answer pairs based on 3D scene object annotations. Each question-answer ...

  65. [73]

    Analyze the properties and spatial relationship of objects in the scene

    Process each question-answer pair, follow these steps: a. Analyze the properties and spatial relationship of objects in the scene. b. Analyze the question and identify question-related objects. c. Construct a logical step-by-step reasoning based on your analyzation to answer t...

  66. [74]

    Don't output anything else

    Output format: '<question_id>;reasoning'. Don't output anything else. Now generate for following annoatations: List description-related objects in description of 3D scene object annotations. Object annotations are structured as: {annotation_id: <annotation_id>, instance_id: <i...

  67. [75]

    Analyze the properties and spatial relationship of objects in the scene based on the annotations

  68. [76]

    Analyze the description and identify description-related objects

    Process each annotation, follow these steps: a. Analyze the description and identify description-related objects. b. Use [<instance_id>] to refer description-related objects

  69. [77]

    Don't output any other things

    Output format: '<annotation_id>;<description-related_objects>'. Don't output any other things. Now generate for following annoatations: Figure 10. The prompt for generating target-relevant objects for ScanQA and ScanRefer. 13 Task Name Instruction Template 3D Reasoning Segment...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.