Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SURPRISE3D is a 200,000-pair 3D benchmark whose human-written spatial queries omit object names, and state-of-the-art 3D grounding and 3D-LLM methods lose substantial accuracy when those shortcuts are removed.

desk verdict A useful 3D spatial reasoning benchmark undercut by an unverified 'shortcut-free' guarantee; the paper deserves review but needs a name-leakage audit and baseline. read the letter →

arxiv 2507.07781 v1 pith:47MTXEIJ submitted 2025-07-10 cs.CV cs.RO

Jiaxin Huang , Ziwen Li , Hanlve Zhang , Runnan Chen , Xiao He , Yandong Guo , Wenping Wang , Tongliang Liu
show 1 more author
Mingming Gong
This is my paper
classification cs.CVcs.RO
keywords spatialreasoning3Dsceneunderstandingvision-languagegroundingreferringsegmentation3D-SRSbenchmarkpointcloudembodiedAI3D-LLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SURPRISE3D, a benchmark built to test whether 3D vision-language models can reason about space instead of recognizing object names. It contributes more than 200,000 vision-language pairs over 900+ indoor ScanNet++ v2 scenes, including 89,000+ human-written spatial queries designed to avoid naming the target object, plus 110,000 LLM-generated common-sense and human-intention queries. The paper defines the 3D Spatial Reasoning Segmentation (3D-SRS) task, in which the model must return a segmentation mask for every object that satisfies a spatial description, and evaluates current expert 3D grounding methods and 3D-LLMs under zero-shot and fine-tuned settings. Those models perform much worse on shortcut-free queries than on conventional grounding benchmarks, with narrative perspective, parametric perspective, and absolute distance as the hardest categories. If the dataset is sound, it provides a way to measure and train spatial understanding separately from object recognition.

What carries the argument

The load-bearing object is the shortcut-free spatial query: a human-written instruction that refers to a target object only through spatial context, not category name, paired with the object's segmentation mask and the camera viewpoint from which the instruction was written. Annotators move to a fixed viewpoint in the scene, write the query as if they were standing there, and draw the target mask directly on the point cloud; multiple annotators then check each query under disambiguation rules that prefer spatial, functional, or visual attributes depending on the situation, and rare-object sampling raises underrepresented classes' query counts by roughly 90-100%. A parallel LLM-plus-human pipeline generates common-sense and intention queries at scale. Any model that succeeds on this material has to parse geometry, reference frames, and distances rather than match words to object categories.

What would settle it

Scan the released 89,000+ human spatial queries for the annotated target object's class name, and run a category-detection baseline that predicts masks from detected object categories alone; if a non-negligible fraction of queries contain the target name, or if the detection baseline solves the task at high accuracy, the claim that the queries force spatial reasoning is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing 3D vision-language benchmarks conflate object recognition with spatial reasoning, because their queries almost always name the target object, letting models score well through category detection. SURPRISE3D removes that crutch: it supplies 200,000+ query-mask pairs across 900+ detailed indoor scenes, with 89,000+ human-written spatial queries that deliberately omit the target object's name and 110,000 LLM-generated queries aimed at common-sense and human-intention reasoning. The spatial queries span relative position, narrative perspective, parametric perspective, and absolute distance, and each one carries a ground-truth segmentation mask plus, for spatial queries, the camera viewpoint that anchors the language. On the proposed 3D-SRS benchmark, current expert 3D grounding methods and 3D-LLMs score much lower on these shortcut-free queries than on name-bearing benchmarks, and fine-tuning on SURPRISE3D lifts their spatial reasoning accuracy by roughly a factor of three.

Load-bearing premise

The entire benchmark rests on the claim that the released human-written spatial queries contain no target-object name and thus cannot be solved by category detection; if even a fraction of them leak the object's name, the shortcut-free premise and the performance gap attributed to missing shortcuts are both undermined, and the paper's own illustrative queries do name the target.

Editorial extensions

If this is right

  • Fine-tuning on SURPRISE3D lifts the spatial reasoning accuracy of current grounding models by roughly three times over their zero-shot scores, so the dataset works as a training resource as well as a diagnostic.
  • A model that passes the 3D-SRS benchmark must compute 3D distances, resolve reference frames, and handle occlusion, which are the same competencies a robot needs to act on spoken spatial instructions.
  • The benchmark splits scores by reasoning type, so future work can report which spatial skills improve and which stay flat instead of relying on a single aggregate number.
  • Because camera parameters are stored with each spatial query, the dataset can be replayed under different viewpoints to test whether models capture viewpoint-dependent language or merely memorize fixed views.
  • The common-sense and human-intention splits extend the evaluation toward pragmatic language, letting embodied agents train on how people actually refer to things rather than on category labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, re-issuing the same spatial queries with object names inserted would quantify the shortcut gap directly: the accuracy difference between named and unnamed versions of identical constraints would measure how much of current 3D grounding is category matching.
  • Beyond the paper's claims, the fixed-viewpoint annotations invite an active-perception variant in which the agent selects its own viewpoint before answering; current static point-cloud models have no mechanism to exploit that freedom.
  • Beyond the paper's claims, a category-detection-only baseline is the missing control; without it, the assertion that the 89,000+ queries resist name-based solving remains untested, and the headline performance gap could partly reflect query ambiguity rather than deficient spatial reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SURPRISE3D, a 3D vision-language dataset built on 900+ ScanNet++ v2 scenes, containing more than 200k query-mask pairs: 89k+ human-written spatial queries and 110k LLM-generated common-sense/intention queries. It defines the 3D Spatial Reasoning Segmentation (3D-SRS) task and reports zero-shot and fine-tuned results for several 3D vision-language models (MLLMfor3D, 3D-Vista, Reason3D, Intent3D, ChatScene). The central claim is that existing models rely on object-name shortcuts and perform substantially worse when queries are crafted without object names, making the benchmark a test of genuine spatial reasoning.

Significance. If the shortcut-free premise is verified, the dataset would be a valuable resource: it is large, mostly human-annotated, covers multi-target queries with segmentation masks, includes rare-object sampling, and introduces a benchmark with reasoning-type breakdowns. The paper's own experiments suggest that fine-tuning on these queries improves spatial reasoning performance, and the authors are to be credited for the human-in-the-loop annotation pipeline, the recording of camera parameters for egocentric queries, and the explicit goal of disentangling spatial reasoning from semantic detection. The paper contains no parametric fitting, so there is no equation-level circularity; the difficulty of the benchmark is partly by construction, which is acceptable for a benchmark paper. However, the headline claim that the human spatial queries are 'deliberately crafted without object name' is load-bearing and, as argued below, is not yet supported by the evidence in the manuscript.

major comments (3)
  1. [Abstract, §1, §3.1, §4.2] The abstract and §1 claim that the 89k+ human-annotated spatial queries are 'deliberately crafted without object name,' and Table 1 lists 'Shortcut Free' as a distinguishing property. However, §1's own examples name the target object ('the vase next to the left door', 'the plamp 2 meters above the floor'), and §3.1's task example is 'The chair closest to the door'. The annotation protocol in §4.2 instructs annotators to 'explicitly nam[e] reference objects' but does not forbid target-category names, and no release-level audit or statistic is provided to show that the actual 89k queries exclude target names. Because the paper's headline conclusion—that low scores reflect missing spatial reasoning—depends on this exclusion, the shortcut-free premise is currently unverified.
  2. [§5.3, Tables 2–3, §5.4] The evaluation mixes incompatible metrics across models under a common 'Average' column: MLLMfor3D and Reason3D produce masks and report mIoU plus Acc, while 3D-Vista, Intent3D, and ChatScene produce boxes and report only Acc. The mIoU rows are averages over only two of the five methods, while the 'Overall' Acc rows average over all five, so cross-row comparisons conflate different metrics and model subsets. In addition, §5.4 states that fine-tuning improves spatial reasoning 'approximately three times,' but the Overall Reasoning A25 figures in Tables 2 and 3 (8.27 vs 18.48) and mIoU (8.89 vs 16.46) imply a factor of about 1.9–2.2, not 3. The quantitative claims need either a single consistent metric (e.g., mask mIoU for all methods, with box outputs converted) or a clear separation of metric-specific analyses.
  3. [§1, §5.1, §5.4] The paper provides no category-detection or text-only baseline that ignores spatial relations (e.g., a model that selects the object whose class name appears in the query, or an open-vocabulary detector). Such a baseline is the minimal control needed to support the claim that the low scores in Table 2 are caused by the absence of object-name shortcuts rather than by the difficulty of the ScanNet++ scenes or the particular annotation distribution. Without it, the experimental section supports only the weaker statement that current models score low on this benchmark, not the stated causal interpretation.
minor comments (6)
  1. [Tables 2–3] The header 'Avarage' should be 'Average' in both tables.
  2. [Abstract and GitHub link] The dataset name and GitHub URL are spelled inconsistently: the abstract and body use 'SURPRISE3D' and 'SUPRISE' in the URL; please standardize the spelling.
  3. [§3.2] The benchmark protocol mentions 'test scenes' for a leaderboard, but the data release only describes train/val splits; clarify how the withheld test set is created and accessed.
  4. [Figure 1] The caption says 'Examples of query categories' but the text does not refer to specific panels; please add panel references when discussing query types.
  5. [§4.2] Rare-object sampling is described as increasing mentions by '∼90–100%' but no formula or baseline is given; report the exact computation or move this figure to the supplementary material.
  6. [§6] The limitations paragraph mentions that 'parametric view' queries may be less natural, but the paper lacks a quantitative breakdown of query-type frequencies and per-type difficulty; adding such a breakdown would strengthen the benchmark analysis.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-level circularity; the 'shortcut-free' premise is a factual validity concern rather than a circular step.

full rationale

This is a dataset and benchmark paper rather than a parametric derivation, so the equation-level circularity patterns do not apply. The headline claim that state-of-the-art 3D grounding methods and 3D-LLMs perform poorly on SURPRISE3D is an empirical measurement on a benchmark that is intentionally difficult; the performance gap is the design goal, not a fitted parameter renamed as a prediction. The only notable self-citation is the authors' own MLLMfor3D baseline (ref [22]), but it is one of several baselines, and the paper's conclusions do not rest solely on that method, so the self-citation is not load-bearing. One caveat is factual rather than circular: the 'shortcut-free' premise—that spatial queries are 'deliberately crafted without object name' (Abstract; Table 1)—is contradicted by the paper's own illustrative queries ('the vase next to the left door', 'the plamp 2 meters above the floor', 'The chair closest to the door') and by the annotation guidance to 'explicitly nam[e] reference objects' (§4.2). This is a correctness/validity concern about the dataset, not a circular derivation, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters, fitted constants, or new physical entities. Its contribution is empirical: the benchmark's validity rests on assumptions about annotation reliability, the enforcement of the shortcut-free property, the fidelity of the underlying ScanNet++ scenes, and the integrity of the data splits. These are domain assumptions rather than mathematically derivable facts.

assumptions (4)
  • domain assumption Human annotations are consistent and unbiased.
    The ground-truth masks and query labels are produced by human annotators with double-checking, but no inter-annotator agreement statistics are reported, so consistency is assumed (Section 4.2).
  • domain assumption Queries labeled 'shortcut-free' contain no target object name.
    The no-object-name property is central to the benchmark's validity, yet the paper's own examples contain object names and no automatic or manual audit is described to guarantee the property across 89K queries (Section 1 and 4.2).
  • domain assumption ScanNet++ v2 reconstructions and recorded camera parameters are accurate enough for grounded spatial reasoning.
    The dataset is built on ScanNet++ v2 scans and uses recorded camera extrinsics/intrinsics to anchor viewpoint-dependent queries; fidelity is inherited from the source dataset (Section 4.1).
  • domain assumption Scene-level splits prevent information leakage.
    The paper says train/validation splits are scene-level, but provides no leakage test or distribution analysis (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes." pith.science (2026). https://pith.science/paper/47MTXEIJ

@misc{pith2026250707781,
  author       = {Pith},
  title        = {Pith review of: SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/47MTXEIJ}},
  note         = {Machine review of arXiv:2507.07781}
}
read the original abstract

The integration of language and 3D perception is critical for embodied AI and robotic systems to perceive, understand, and interact with the physical world. Spatial reasoning, a key capability for understanding spatial relationships between objects, remains underexplored in current 3D vision-language research. Existing datasets often mix semantic cues (e.g., object name) with spatial context, leading models to rely on superficial shortcuts rather than genuinely interpreting spatial relationships. To address this gap, we introduce S\textsc{urprise}3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes. S\textsc{urprise}3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2, including more than 2.8k unique object classes. The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name, thereby mitigating shortcut biases in spatial understanding. These queries comprehensively cover various spatial reasoning skills, such as relative position, narrative perspective, parametric perspective, and absolute distance reasoning. Initial benchmarks demonstrate significant challenges for current state-of-the-art expert 3D visual grounding methods and 3D-LLMs, underscoring the necessity of our dataset and the accompanying 3D Spatial Reasoning Segmentation (3D-SRS) benchmark suite. S\textsc{urprise}3D and 3D-SRS aim to facilitate advancements in spatially aware AI, paving the way for effective embodied interaction and robotic planning. The code and datasets can be found in https://github.com/liziwennba/SUPRISE.

Figures

Figures reproduced from arXiv: 2507.07781 by the authors.

Figure 1
Figure 1. Examples of query categories in our 3D-SRS task. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Spatial reasoning annotation pipeline. Human Annotators select a scene and target object, then write a question that identifies the object via spatial context, and finally mask the object. Absolute Distance-based reasoning: interpreting absolute or comparative distance cues. Many queries involve terms such as ’closest’, ’furthest’, or ’near’. The model should be able to compare distances between multiple objects wit… view at source ↗
Figure 3
Figure 3. Overview of the common sense and human intention reasoning queries generation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: SURPRISE3D Dataset Statistics and Enhancements [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The comparison between zero-shot and fine-tuned models on all reasoning tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. G$^2$TAM: Geometry Grounded Track Anything Model

    cs.CV 2026-07 accept novelty 6.5 of 10

    Spatially aligned geometric features serve as implicit memory so one model reconstructs scenes and produces promptable, cross-view consistent instance masks from unordered RGB only.

  2. Holo-Captioning: Toward the Text Equivalent of 3D Scenes

    cs.CV 2026-07 conditional novelty 6.5 of 10

    HoloScribe generates pure-text structured captions of all entities, boxes, attributes and relations in 3D indoor scenes and outperforms dense captioners and 3D LLMs on a new 15K-scene benchmark.

  3. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5 of 10

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

Reference graph

Works this paper leans on

47 extracted references · 34 canonical work pages · cited by 3 Pith papers

  1. [1]

    Scanents3d: Exploiting phrase-to-3d-object correspondences for improved visio-linguistic models in 3d scenes.WACV, 2022

    Ahmed Abdelreheem, Kyle Olszewski, Hsin-Ying Lee, Peter Wonka, and Panos Achlioptas. Scanents3d: Exploiting phrase-to-3d-object correspondences for improved visio-linguistic models in 3d scenes.WACV, 2022. 9

    work page 2022

  2. [2]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, pages 422–440. Springer, 2020

    work page 2020

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. InCVPR, 2022

    2022

  4. [4]

    Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, et al. Cosmos-reason1: From physical common sense to embodied reasoning.arXiv preprint arXiv:2503.15558, 2025

    arXiv 2025

  5. [5]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InECCV, pages 202–221. Springer, 2020

    work page 2020

  6. [6]

    Dave Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X. Chang. Scan2cap: Context- aware dense captioning in rgb-d scans, 2020

    work page 2020

  7. [7]

    Towards label-free scene understanding by vision foundation models

    Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, and Wenping Wang. Towards label-free scene understanding by vision foundation models. InNeurIPS, pages 75896–75910, 2023

    work page 2023

  8. [8]

    Clip2scene: Towards label-efficient 3d scene understanding by clip

    Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. InCVPR, pages 7020–7030, 2023

    work page 2023

Show all 47 references
  1. [9]

    Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies.arXiv preprint arXiv:2501.00326, 2024

    Runnan Chen, Xiangyu Sun, Zhaoqing Wang, Youquan Liu, Jiepeng Wang, Lingdong Kong, Jiankang Deng, Mingming Gong, Liang Pan, Wenping Wang, et al. Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies.arXiv preprint arXiv:2501.00326, 2024

    2024 arXiv

  2. [10]

    Zero-shot point cloud segmentation by transferring geometric primitives.arXiv preprint arXiv:2210.09923, 2022

    Runnan Chen, Xinge Zhu, Nenglun Chen, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang. Zero-shot point cloud segmentation by transferring geometric primitives.arXiv preprint arXiv:2210.09923, 2022

    2022 arXiv

  3. [11]

    Bridging language and geometric primitives for zero-shot point cloud segmentation

    Runnan Chen, Xinge Zhu, Nenglun Chen, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang. Bridging language and geometric primitives for zero-shot point cloud segmentation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5380–5388, 2023

    2023

  4. [12]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InCVPR, pages 26428–26438, 2024

    2024

  5. [13]

    Reasoning3d–grounding and reasoning in 3d: Fine-grained zero-shot open-vocabulary 3d reasoning part segmentation via large vision-language models

    Tianrun Chen, Chunan Yu, Jing Li, Jianqi Zhang, Lanyun Zhu, Deyi Ji, Yong Zhang, Ying Zang, Zejian Li, and Lingyun Sun. Reasoning3d–grounding and reasoning in 3d: Fine-grained zero-shot open-vocabulary 3d reasoning part segmentation via large vision-language models. arXiv prep...

    2024 arXiv

  6. [14]

    Grounded 3d-llm with referent tokens.arXiv preprint arXiv: 2405.10370, 2024

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens.arXiv preprint arXiv: 2405.10370, 2024

    2024 arXiv

  7. [15]

    Yasaman Etesam, Leon Kochiev, and Angel X. Chang. 3dvqa: Visual question answering for 3d environments. InConference on Robots and Vision (CRV), 2022

    2022

  8. [16]

    Segpoint: Segment any point cloud via large language model

    Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. Segpoint: Segment any point cloud via large language model. InECCV, pages 349–367. Springer, 2024

    2024

  9. [17]

    3d concept learning and reasoning from multi-view images

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. InCVPR, pages 9202–9212, 2023

    2023

  10. [18]

    3d-llm: injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: injecting the 3d world into large language models. InNeurIPS, pages 20482–20494, 2023. 10

    2023

  11. [19]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. InNeurIPS, 2024

    2024

  12. [20]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. InICML, pages 20413–20451, 2024

    2024

  13. [21]

    Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation.arXiv preprint arXiv:2503.18135, 2025

    Jiaxin Huang, Runnan Chen, Ziwen Li, Zhengqing Gao, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation.arXiv preprint arXiv:2503.18135, 2025

    2025

  14. [22]

    Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation, 2025

    Jiaxin Huang, Runnan Chen, Ziwen Li, Zhengqing Gao, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation, 2025

    2025

  15. [23]

    Rea- son3d: Searching and reasoning 3d segmentation via large language model.arXiv preprint arXiv:2405.17427, 2024

    Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Rea- son3d: Searching and reasoning 3d segmentation via large language model.arXiv preprint arXiv:2405.17427, 2024

    2024 arXiv

  16. [24]

    Text-guided graph neural networks for referring 3d instance segmentation.AAAI, 35(2):1610–1618, May 2021

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation.AAAI, 35(2):1610–1618, May 2021

    2021

  17. [25]

    Dense object grounding in 3d scenes.ACM Multimedia, 2023

    Wencan Huang, Daizong Liu, and Wei Hu. Dense object grounding in 3d scenes.ACM Multimedia, 2023

    2023

  18. [26]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. InECCV, pages 289–310. Springer, 2024

    2024

  19. [27]

    Multimodal 3d reasoning segmentation with complex scenes.arXiv preprint arXiv: 2411.13927, 2024

    Xueying Jiang, Lewei Lu, Ling Shao, and Shijian Lu. Multimodal 3d reasoning segmentation with complex scenes.arXiv preprint arXiv: 2411.13927, 2024

    2024 arXiv

  20. [28]

    Intent3d: 3d object detection in rgb-d scans based on human intention

    Weitai Kang, Mengxue Qu, Jyoti Kini, Yunchao Wei, Mubarak Shah, and Yan Yan. Intent3d: 3d object detection in rgb-d scans based on human intention. InICLR, 2025

    2025

  21. [29]

    M3dbench: Towards omni 3d assistant with interleaved multi-modal instructions

    Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Zhuoyuan Li, Gang Yu, and Tao Chen. M3dbench: Towards omni 3d assistant with interleaved multi-modal instructions. InECCV, pages 41–59. Springer, 2024

    2024

  22. [30]

    3dmit: 3d multi-modal instruction tuning for scene understanding.arXiv preprint arXiv:2401.03201, 2024

    Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, and Xiangde Liu. 3dmit: 3d multi-modal instruction tuning for scene understanding.arXiv preprint arXiv:2401.03201, 2024

    2024 arXiv

  23. [31]

    Multi-modal situated reasoning in 3d scenes.NeurIPS, 2024

    Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes.NeurIPS, 2024

    2024

  24. [32]

    See more and know more: Zero-shot point cloud segmentation via multi-modal visual data

    Yuhang Lu, Qi Jiang, Runnan Chen, Yuenan Hou, Xinge Zhu, and Yuexin Ma. See more and know more: Zero-shot point cloud segmentation via multi-modal visual data. InICCV, pages 21674–21684, 2023

    2023

  25. [33]

    3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825, 2024

    Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: A comprehensive 3d spatial reasoning benchmark.arXiv preprint arXiv:2412.07825, 2024

    2024

  26. [34]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. InICLR, 2023

    2023

  27. [35]

    X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks

    Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. InAAAI, volume 38, pages 4551–4559, 2024

    2024

  28. [36]

    Fully convolutional networks for semantic segmentation.IEEE TPAMI, 39(4):640–651, 2017

    Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation.IEEE TPAMI, 39(4):640–651, 2017. 11

    2017

  29. [37]

    Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. InCVPR, 2024

    2024

  30. [38]

    Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    2023 arXiv

  31. [39]

    3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation

    Changli Wu, Yiwei Ma, Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation. InAAAI, volume 38, pages 5940–5948, 2024

    2024

  32. [40]

    Com- prehensive visual question answering on point clouds through compositional scene manipulation, 2023

    Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo, Zhen Li, and Shuguang Cui. Com- prehensive visual question answering on point clouds through compositional scene manipulation, 2023

    2023

  33. [41]

    Fouhey, and Joyce Chai

    Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David F. Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination, 2024

    2024

  34. [42]

    Scannet++: A high-fidelity dataset of 3d indoor scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. InICCV, pages 12–22, 2023

    2023

  35. [43]

    Excap3d: Expressive 3d scene understanding via object captioning with varying detail.arXiv preprint arXiv: 2503.17044, 2025

    Chandan Yeshwanth, David Rozenberszki, and Angela Dai. Excap3d: Expressive 3d scene understanding via object captioning with varying detail.arXiv preprint arXiv: 2503.17044, 2025

    2025 arXiv

  36. [44]

    Toward explainable and fine-grained 3d grounding through referring textual phrases.arXiv preprint arXiv:2207.01821, 2022

    Zhihao Yuan, Xu Yan, Zhuo Li, Xuhao Li, Yao Guo, Shuguang Cui, and Zhen Li. Toward explainable and fine-grained 3d grounding through referring textual phrases.arXiv preprint arXiv:2207.01821, 2022

    2022 arXiv

  37. [45]

    Vla-3d: A dataset for 3d semantic scene understanding and navigation.arXiv preprint arXiv: 2411.03540, 2024

    Haochen Zhang, Nader Zantout, Pujith Kachana, Zongyuan Wu, Ji Zhang, and Wenshan Wang. Vla-3d: A dataset for 3d semantic scene understanding and navigation.arXiv preprint arXiv: 2411.03540, 2024

    2024 arXiv

  38. [46]

    Scanreason: Empowering 3d visual grounding with reasoning capabilities.arXiv preprint arXiv: 2407.01525, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities.arXiv preprint arXiv: 2407.01525, 2024

    2024 arXiv

  39. [47]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InICCV, pages 2911–2921, 2023. 12

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.