Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims to be the first systematic review of 3D Scene Question Answering, organizing the field into datasets, methods, and evaluation metrics, and revealing a shared architecture behind nearly all systems.

desk verdict Useful survey of 3D SQA, but the central comparison tables have inconsistencies and a suspiciously duplicated row that need fixing before the numbers can be trusted. read the letter →

arxiv 2502.00342 v2 pith:GDMF63JF submitted 2025-02-01 cs.CV

Zechuan Li , Hongshan Yu , Yihao Ding , Yan Li , Yong He , Naveed Akhtar This is my paper · ORCID
classification cs.CV
keywords 3DSceneQuestionAnsweringVisualPerceptionMultimodalityLargeLanguageModelInstruction-tuningZero-shotSurveyEmbodiedIntelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a survey of 3D Scene Question Answering (3D SQA), the task of answering questions about a 3D environment from point clouds, multi-view images, or both. It claims to be the first systematic review of the field, organizing it along three axes: datasets, methods, and evaluation metrics. The survey's central synthesis is that nearly all existing 3D SQA systems share one architectural pattern — modality-specific encoders feeding a fusion module and a prediction head — and that the field has moved from closed-set task-specific models to pretraining-based instruction tuning and zero-shot inference. If the survey's mapping is accurate, it gives researchers a reliable map for choosing datasets and baselines, and it identifies instruction-tuned pretrained models as the current best-performing approach while zero-shot methods lag. The review also catalogs open challenges such as fragmented benchmarks, LLM-generated data quality, and the lack of a unified evaluation protocol.

What carries the argument

The organizing device is a three-perspective taxonomy plus a generic architecture. The generic architecture is a pipeline in which a 3D scene (point cloud $S^{(p)}$ or multi-view images $S^{(m)}$) and a query (text, egocentric view, or object fragments) are encoded separately, fused by a transformer-style module, and passed to a head that outputs an answer text $T$ and optional 3D bounding boxes $B^{(3D)}$. The taxonomy classifies methods as task-specific (closed-set classification), pretraining-based (with or without instruction tuning), and zero-shot (text-driven, image-driven, or multimodal alignment). This machinery lets the survey compare otherwise heterogeneous papers under a common lens and is what supports its claims about field-level trends.

What would settle it

A reader could falsify the central claim by checking Table 4's modality labels against the original papers — for instance, 3D-LLM is listed with image-only scene modality while its own paper describes multimodal inputs — and by searching the literature for a 3D SQA survey published before 2025.

Watch

Extended reading notes

Core claim

The central claim is that 3D SQA can be systematically understood as a co-evolution of datasets, methods, and metrics, and that this review is the first to lay that structure out. On datasets, the survey distinguishes synthetic, point-cloud, multi-view, and multimodal collections, and traces QA-pair generation from templates and human annotation to LLM-assisted pipelines. On methods, it proposes a three-part taxonomy — task-specific, pretraining-based, and zero-shot — and extracts a shared pipeline of scene encoder, query encoder, fusion module, and prediction head with optional 3D bounding box grounding. On evaluation, it contrasts exact-match and n-gram metrics with LLM-based semantic scoring. The survey's comparative tables show instruction-tuned pretraining methods leading on ScanQA and SQA3D, zero-shot methods trailing, and all methods well below human accuracy.

Load-bearing premise

The survey's usefulness rests on its categorization tables and its claim to be the first review; if some datasets or methods are mislabeled, or if an earlier survey exists, the systematic comparison loses its reliability.

Editorial extensions

If this is right

  • If the survey's mapping holds, new 3D SQA work can be positioned quickly: datasets by modality and query type, methods by taxonomy class, and results against the ScanQA and SQA3D leaderboards.
  • Instruction-tuned pretraining-based models, such as Scene-LLM and Spartun3D-LLM, are currently the strongest performers on both benchmarks, so future method development should assume this as the baseline to beat.
  • Zero-shot methods are not yet competitive for fine-grained spatial reasoning, implying that improving explicit 3D representation in zero-shot pipelines is a concrete research target.
  • The absence of a unified evaluation protocol is a real obstacle: scores across datasets are not directly comparable, and LLM-based metrics are emerging as a complement to exact match and n-gram scores.
  • All current systems remain far below human performance on open-ended and multi-step reasoning, so the field has clear headroom.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The shared pipeline identified by the survey suggests that scene encoders and fusion modules may transfer across methods; one testable extension is whether a standard point-cloud encoder plus an instruction-tuned LLM matches specialized designs on ScanQA.
  • The survey's 'first' claim could be checked by searching for earlier 3D SQA overviews; if one exists, the contribution shifts from novelty to consolidation.
  • The LLM-based metric family, illustrated by LLM-Match, could become the default for open-ended 3D SQA, and a natural next step is a standardized prompt and scoring rubric across benchmarks.
  • The dataset-quality problem in LLM-generated QA pairs suggests an opportunity for automated validation tools that check spatial consistency, which the survey lists as a needed direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents a survey of 3D Scene Question Answering (3D SQA), organizing the field along three axes: datasets, evaluation metrics, and methods. It introduces a formal task notation, taxonomizes existing datasets by scene and query modality, categorizes methods into task-specific, pretraining-based, and zero-shot approaches, and draws conclusions about current state-of-the-art performance on the ScanQA and SQA3D benchmarks. The survey also identifies open challenges and future research directions. The central claim is that this is the first comprehensive and systematic review of 3D SQA, and that the proposed categorization reveals shared architectural patterns across methods.

Significance. If the survey's categorization and benchmark tables are reliable, the paper would serve as a useful entry point for researchers in 3D SQA, aggregating dataset statistics, method characteristics, and performance numbers in one place. The paper covers a broad range of recent work, including instruction-tuned and zero-shot LVLM-based methods, and its discussion of evaluation metrics and open problems is relevant to the community. The attempt to identify a common architectural pipeline across task-specific, pretraining, and zero-shot methods is also valuable. However, the utility of the survey depends entirely on the accuracy and internal consistency of its comparison tables, and several load-bearing entries in Tables 2, 4, 5, and 6 are internally contradictory or implausible. Until these are corrected and verified, the systematic claims in Sections 5.1–5.4 cannot be accepted as a faithful map of the field.

major comments (4)
  1. [Table 4 (3D-LLM row) and Table 2 (3D-LLM row)] The modality labels for 3D-LLM are contradictory: Table 2 lists 3D-LLM with scene modality {S(m), S(p)} (multimodal), while Table 4 lists the same work with scene modality S(m) (image-only). The text in Section 5.2 further describes 3D-LLM as building on BLIP2, which does not clarify the 3D input. Since the original paper consumes point clouds as well as rendered images, one of the two labels is wrong. This inconsistency matters because Section 5 uses method-modality assignments to identify shared architectural patterns, and a flagship method's modality affects the claimed pattern. The authors must verify the original source and make the two tables consistent.
  2. [Table 5, ESZG row] The ESZG row in Table 5 reports BLEU-1..4 = 30.24/20.40/15.11/10.08, ROUGE = 33.33, METEOR = 13.14, CIDEr = 64.86, and EM@10 = 18.01. The ScanQA baseline row in the same table reports nearly identical n-gram statistics (BLEU-1..4 = 30.24/20.40/15.11/10.08, ROUGE = 33.30, METEOR = 13.14, CIDEr = 64.90). It is implausible that an independent zero-shot GPT-4V evaluation reproduces the closed-set baseline's corpus-level n-gram scores to three significant digits while differing in EM@10 by more than a factor of two. As printed, this row appears to have been copied from the baseline, and Section 5.4's conclusion that zero-shot methods perform worst relies on this table. The authors must source-verify the ESZG numbers or remove the row.
  3. [Table 4 and Table 6, LEO, LAMM, and Spartun3D-LLM rows] Several methods are assigned to mutually exclusive categories. LEO appears twice in Table 4, once as P-B(w I-T) and once as Z-S, and Spartun3D-LLM appears as Z-S in Table 4 but as P-B(w I-T) in Table 6; LAMM also appears in both the P-B(w I-T) and Z-S blocks of Table 4. The text in Section 5.3 justifies treating LEO and Spartun3D-LLM as zero-shot methods 'for inference' after pretraining, but this conflates 'zero-shot evaluation of a fine-tuned model' with 'zero-shot learning method' as defined in the taxonomy. Either the category definitions need to be refined to accommodate this dual use, or the table entries must be corrected. As written, the taxonomy in Section 5 is not mutually exclusive and undermines the comparative claims made from it.
  4. [Section 3.1.1 and Table 2, CLEVR3D citations] CLEVR3D is cited inconsistently: Section 3.1.1 says "CLEVR3D [43] utilized functional programs" while Table 2 and Section 3.2.1 cite CLEVR3D as [102]. Reference [43] is Johnson et al.'s 2D CLEVR paper, not the 3D point-cloud CLEVR3D dataset (Yan et al., [102]). This is not merely a formatting slip; it misattributes the dataset to the wrong publication. All citations to CLEVR3D should be checked and unified.
minor comments (6)
  1. [Section 3.2.2] The text uses "3D VQA questions" and "3D SQA" interchangeably; please standardize the terminology to avoid confusion between scene question answering and the narrower visual question answering task.
  2. [Table 5 and Table 4, SceneVerse/GPS] Table 5 lists "SceneVerse [42]" while Table 4 lists "GPS [42]" in the pretraining block; if these refer to the same paper or method, the names and type labels should be aligned to prevent ambiguity.
  3. [Section 5.4] The text says "Tables 5 and 6 in the appendix" but both tables appear in the main body; please correct the cross-reference.
  4. [Table 2, 'ROBOSPATIAL' vs 'Robospatial'] The dataset is written as "ROBOSPATIAL" in Table 2 and "Robospatial" in Section 3.1.1; use a single consistent spelling.
  5. [Section 4.1] SPICE is defined among the traditional metrics but is not reported in any of the comparison tables (Tables 5 and 6); either include it where available or note that it is not used in the surveyed evaluations.
  6. [Figure 2] The hierarchical diagram in Figure 2 is visually dense and the font is very small; enlarging or splitting it into separate figures would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey reports and categorizes external literature, performs no derivations or fits, and its self-citations are background references, not load-bearing premises.

full rationale

This paper is a survey of 3D Scene Question Answering. It contains no derivation chain: it does not fit parameters, define quantities in terms of each other, or claim to predict a result from its own outputs. Its central claims are organizational ('we provide the first comprehensive and systematic review of 3D SQA', 'we organize existing work from three perspectives') and are about the cited literature, not about internal inference. The self-citations present in the reference list (e.g., [35], [50], [119]) are background citations to the authors' prior work on 3D segmentation, detection, and point-cloud processing; none of these is invoked as the justification for a load-bearing premise or as a uniqueness theorem, and none is used to forbid alternatives. The paper's stated limitations, such as the lack of unified evaluation protocols and the difficulties of dataset standardization, are explicitly discussed rather than hidden. Internal inconsistencies in the comparison tables (e.g., the differing modality labels for 3D-LLM in Tables 2 and 4, and the near-identical n-gram scores between the ScanQA baseline and ESZG in Table 5) are potential correctness or transcription issues that undermine the reliability of the survey's map, but they are not instances of circular reasoning: the survey is not defining its conclusions in terms of its own inputs, nor is it renaming a known result as a prediction. Under the stated criteria, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new entities, parameters, or derivations introduced; this is a survey. The assumptions listed are the interpretive choices the survey makes when organizing the literature.

assumptions (3)
  • domain assumption The selected taxonomy (datasets vs methods vs metrics, and T-S/P-B/Z-S categories) is exhaustive and disjoint.
    Central organization of the survey; no formal justification that categories cover all 3D SQA work or that methods cannot belong to multiple types.
  • domain assumption Performance numbers in Tables 5 and 6 are accurately transcribed from the cited papers.
    The survey aggregates results from many papers without re-running experiments; errors in transcription propagate into conclusions about state-of-the-art.
  • domain assumption The field as defined, 3D SQA, is a coherent task with comparable evaluation frameworks across point cloud and multi-view representations.
    Defines the task via Eq. 3; underlying assumption that point-cloud and multi-view representations share enough structure to be organized in one survey.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering." pith.science (2026). https://pith.science/paper/GDMF63JF

@misc{pith2026250200342,
  author       = {Pith},
  title        = {Pith review of: Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDMF63JF}},
  note         = {Machine review of arXiv:2502.00342}
}
read the original abstract

3D Scene Question Answering (3D SQA) represents an interdisciplinary task that integrates 3D visual perception and natural language processing, empowering intelligent agents to comprehend and interact with complex 3D environments. Recent advances in large multimodal modelling have driven the creation of diverse datasets and spurred the development of instruction-tuning and zero-shot methods for 3D SQA. However, this rapid progress introduces challenges, particularly in achieving unified analysis and comparison across datasets and baselines. In this survey, we provide the first comprehensive and systematic review of 3D SQA. We organize existing work from three perspectives: datasets, methodologies, and evaluation metrics. Beyond basic categorization, we identify shared architectural patterns across methods. Our survey further synthesizes core limitations and discusses how current trends, such as instruction tuning, multimodal alignment, and zero-shot, can shape future developments. Finally, we propose a range of promising research directions covering dataset construction, task generalization, interaction modeling, and unified evaluation protocols. This work aims to serve as a foundation for future research and foster progress toward more generalizable and intelligent 3D SQA systems.

Figures

Figures reproduced from arXiv: 2502.00342 by the authors.

Figure 1
Figure 1. 2D Scene VQA and 3D SQA tasks. 3D SQA handles non￾embodied as well as embodied tasks involving agent interactions within 3D scenes. To analyse the emerging challenges in 3D SQA and facilitate their systematic handling, this paper provides the first comprehensive survey of this research direction. We focus on three fundamental aspects of this area, namely; (i) the objectives of 3D SQA, (ii) datasets needed to support… view at source ↗
Figure 2
Figure 2. Graphical illustration of the hierarchical structure of 3D SQA literature adopted in this work. A systematic categorization is adopted for preliminaries, datasets, evaluation metrics and methodologies. where each 𝐱𝑖 represents a 3D coordinate. The multi-view RGB image input is represented as: 𝑆 (𝑚) = {𝐼1 , 𝐼2 ,…, 𝐼𝐾}. (5) The query 𝑄 may also be composed of multiple modali￾ties. A textual question is represented as … view at source ↗
Figure 3
Figure 3. Dataset generation workflow. scenarios as they lack description of an agent’s spatial or contextual relationship with the scene. Consequently, these datasets are limited in scope, as reflected in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of a generalized 3D SQA pipeline. The scene input—represented as images or point clouds—is processed by a visual encoder, while the question input—comprising textual and potentially egocentric visual components—is encoded separately. The resulting features are…
Figure 5
Figure 5. Figure 5: Typical architecture of task-specific 3D SQA methods. Scene and query (question) features are encoded separately, fused via a transformer-based module, and used to predict the answer, optionally with bounding boxes and object categories. multimodal understanding by the…
Figure 6
Figure 6. Figure 6: Illustration of an instruction-tuned 3D SQA framework. The system encodes multimodal inputs—including text, multi-view images, and point clouds—into a shared embedding space. Through alignment modules and fusion strategies, these features are combined with task-specifi…
Figure 7
Figure 7. Figure 7: Illustration of three representative paradigms for zero-shot 3D SQA. Left: Text-Driven methods (e.g., MLLM-based) use scene descriptions obtained from human annotations or external datasets, which are processed together with user queries via LLMs. Middle: Image-Driven …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

Reference graph

Works this paper leans on

132 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [43]

    Clevr:Adiagnosticdataset for compositional language and elementary visual reasoning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., LawrenceZitnick,C.,Girshick,R.,2017. Clevr:Adiagnosticdataset for compositional language and elementary visual reasoning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901–2910

    2017

  2. [102]

    Yan, X., Yuan, Z., Du, Y., Liao, Y., Guo, Y., Cui, S., Li, Z.,

    work page

  3. [1]

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.,

  4. [2]

    Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., Guibas, L., 2020. Referit3d: Neural listeners for fine-grained 3d object identificationinreal-worldscenes,in:ComputerVision–ECCV2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, Springer. pp. 422–440

    2020

  5. [3]

    Anderson, P., Fernando, B., Johnson, M., Gould, S., 2016. Spice: Semantic propositional image caption evaluation, in: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, Springer. pp. 382–398

    2016

  6. [4]

    Vqa: Visual question answering, in: Proceedings of the IEEE international conference on computer vision, pp

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D., 2015. Vqa: Visual question answering, in: Proceedings of the IEEE international conference on computer vision, pp. 2425– 2433

    2015

  7. [5]

    Circle: Capture in rich contextual environments, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Araújo, J.P., Li, J., Vetrivel, K., Agarwal, R., Wu, J., Gopinath, D., Clegg, A.W., Liu, K., 2023. Circle: Capture in rich contextual environments, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21211–21221

    2023

  8. [6]

    Scanqa: 3d question answering for spatial scene understanding, in: proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Azuma, D., Miyanishi, T., Kurita, S., Kawanabe, M., 2022. Scanqa: 3d question answering for spatial scene understanding, in: proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129–19139

    2022

Show all 132 references
  1. [7]

    Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery

    Bai, L., Wang, G., Islam, M., Seenivasan, L., Wang, A., Ren, H., 2025. Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery. Information Fusion 113, 102602

    2025

  2. [8]

    Banerjee, S., Lavie, A., 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72

    2005

  3. [9]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene under- standing using mobile rgb-d data

    Baruch, G., Chen, Z., Dehghan, A., Dimry, T., Feigin, Y., Fu, P., Gebauer, T., Joffe, B., Kurz, D., Schwartz, A., et al., 2021. Arkitscenes: A diverse real-world dataset for 3d indoor scene under- standing using mobile rgb-d data. arXiv preprint arXiv:2111.08897

    2021 arXiv

  4. [10]

    Models for multiparty engagement in open-worlddialog,in:ProceedingsoftheSIGDIAL2009conference, the 10th annual meeting of the special interest group on discourse and dialogue, p

    Bohus, D., Horvitz, E., 2009. Models for multiparty engagement in open-worlddialog,in:ProceedingsoftheSIGDIAL2009conference, the 10th annual meeting of the special interest group on discourse and dialogue, p. 10

    2009

  5. [11]

    Language models are few-shot learners

    Brown, T.B., 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    2020 arXiv

  6. [12]

    Scanrefer: 3d object localization in rgb-d scans using natural language, in: European conference on computer vision, Springer

    Chen, D.Z., Chang, A.X., Nießner, M., 2020. Scanrefer: 3d object localization in rgb-d scans using natural language, in: European conference on computer vision, Springer. pp. 202–221

    2020

  7. [13]

    Ll3da: Visual interactive instruction tuning for omni-3dunderstandingreasoningandplanning,in:Proceedingsofthe IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T., 2024. Ll3da: Visual interactive instruction tuning for omni-3dunderstandingreasoningandplanning,in:Proceedingsofthe IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp. 26428–26438

    2024

  8. [14]

    Lan- guage conditioned spatial relation reasoning for 3d object grounding

    Chen,S.,Guhur,P.L.,Tapaswi,M.,Schmid,C.,Laptev,I.,2022. Lan- guage conditioned spatial relation reasoning for 3d object grounding. Advancesinneuralinformationprocessingsystems35,20522–20535

    2022

  9. [15]

    End-to- end 3d dense captioning with vote2cap-detr, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., Chen, T., 2023. End-to- end 3d dense captioning with vote2cap-detr, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11124–11133

    2023

  10. [16]

    Scan2cap: Context-awaredensecaptioninginrgb-dscans,in:Proceedingsofthe IEEE/CVF conference on computer vision and pattern recognition, pp

    Chen, Z., Gholami, A., Nießner, M., Chang, A.X., 2021. Scan2cap: Context-awaredensecaptioninginrgb-dscans,in:Proceedingsofthe IEEE/CVF conference on computer vision and pattern recognition, pp. 3193–3203

    2021

  11. [17]

    Vicuna: An open-sourcechatbotimpressinggpt-4with90%*chatgptquality

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al., 2023. Vicuna: An open-sourcechatbotimpressinggpt-4with90%*chatgptquality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2, 6

    2023

  12. [18]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    Dai,A.,Chang,A.X.,Savva,M.,Halber,M.,Funkhouser,T.,Nießner, M., 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828–5839

    2017

  13. [19]

    Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., Batra, D.,

  14. [20]

    Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.,

  15. [21]

    Multi-clip: Contrastive vision- languagepre-trainingforquestionansweringtasksin3dscenes.arXiv preprint arXiv:2306.02329

    Delitzas, A., Parelli, M., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., Hofmann, T., 2023. Multi-clip: Contrastive vision- languagepre-trainingforquestionansweringtasksin3dscenes.arXiv preprint arXiv:2306.02329

    2023 arXiv

  16. [22]

    Ding, Y., Ren, K., Huang, J., Luo, S., Han, S.C., 2024. Mmvqa: A comprehensive dataset for investigating multipage multimodal informationretrievalinpdf-basedvisualquestionanswering,in:33rd International Joint Conference on Artificial Intelligence, IJCAI 2024, International Joi...

    2024

  17. [23]

    13142–13153

    Objaverse:Auniverseofannotated3dobjects,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142–13153

  18. [24]

    1billion: A large-scale benchmark for general object grasping

    Fang, H., Wang, C., Gou, M., Lu, C.G., . 1billion: A large-scale benchmark for general object grasping. in 2020 ieee, in: CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11441–11450

    2020

  19. [25]

    Scene-llm: Extendinglanguagemodelfor3dvisualunderstandingandreasoning

    Fu, R., Liu, J., Chen, X., Nie, Y., Xiong, W., 2024. Scene-llm: Extendinglanguagemodelfor3dvisualunderstandingandreasoning. arXiv preprint arXiv:2403.11401

    2024 arXiv

  20. [26]

    3dvqa: Visual question answering for 3d environments, in: 2022 19th Conference on Robots and Vision (CRV), pp

    Etesam, Y., Kochiev, L., Chang, A.X., 2022. 3dvqa: Visual question answering for 3d environments, in: 2022 19th Conference on Robots and Vision (CRV), pp. 233–240. doi:10.1109/CRV55824.2022.00038

    2022

  21. [27]

    Long short-term memory

    Graves, A., Graves, A., 2012. Long short-term memory. Supervised sequence labelling with recurrent neural networks , 37–45

    2012

  22. [28]

    Deep learning for 3d point clouds: A survey

    Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., Bennamoun, M., 2020. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence 43, 4338–4364

    2020

  23. [29]

    Iqa: Visual question answering in interactive environments, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., Farhadi, A., 2018. Iqa: Visual question answering in interactive environments, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4089–4098

    2018

  24. [30]

    Stochastic scene-aware motion prediction, in: ProceedingsoftheIEEE/CVFInternationalConferenceonComputer Vision, pp

    Hassan, M., Ceylan, D., Villegas, R., Saito, J., Yang, J., Zhou, Y., Black, M.J., 2021. Stochastic scene-aware motion prediction, in: ProceedingsoftheIEEE/CVFInternationalConferenceonComputer Vision, pp. 11374–11384

    2021

  25. [31]

    Resolving 3dhumanposeambiguitieswith3dsceneconstraints,in:Proceedings of the IEEE/CVF international conference on computer vision, pp

    Hassan, M., Choutas, V., Tzionas, D., Black, M.J., 2019. Resolving 3dhumanposeambiguitieswith3dsceneconstraints,in:Proceedings of the IEEE/CVF international conference on computer vision, pp. 2282–2292

    2019

  26. [32]

    A review of algorithms for filtering the 3d point cloud

    Han, X.F., Jin, J.S., Wang, M.J., Jiang, W., Gao, L., Xiao, L., 2017. A review of algorithms for filtering the 3d point cloud. Signal Processing: Image Communication 57, 103–112

    2017

  27. [33]

    Mask r-cnn, in: ProceedingsoftheIEEEinternationalconferenceoncomputervision, pp

    He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017. Mask r-cnn, in: ProceedingsoftheIEEEinternationalconferenceoncomputervision, pp. 2961–2969

    2017

  28. [34]

    Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Z. Li et al. :Preprint submitted to Elsevier Page 16 of 19 Embodied Intelligence for 3D Unde...

    2016

  29. [35]

    Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding, in: Proceedings of the 29th ACM International Conference on Multimedia, pp

    He,D.,Zhao,Y.,Luo,J.,Hui,T.,Huang,S.,Zhang,A.,Liu,S.,2021. Transrefer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding, in: Proceedings of the 29th ACM International Conference on Multimedia, pp. 2344–2352

    2021

  30. [36]

    Long short-term memory

    Hochreiter, S., 1997. Long short-term memory. Neural Computation MIT-Press

    1997

  31. [37]

    3d concept learning and reasoning from multi-view images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Hong, Y., Lin, C., Du, Y., Chen, Z., Tenenbaum, J.B., Gan, C., 2023a. 3d concept learning and reasoning from multi-view images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9202–9212

  32. [38]

    Deep learning based 3d segmentation in computer vision: A survey

    He,Y.,Yu,H.,Liu,X.,Yang,Z.,Sun,W.,Anwar,S.,Mian,A.,2025. Deep learning based 3d segmentation in computer vision: A survey. Information Fusion 115, 102722

    2025

  33. [39]

    Chat-scene: Bridging 3d scene and large language models with object identifiers, in: The Thirty-eighth Annual Conference on Neural Information Processing Systems

    Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al., 2024. Chat-scene: Bridging 3d scene and large language models with object identifiers, in: The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  34. [40]

    An embodied generalist agent in 3d world

    Huang,J.,Yong,S.,Ma,X.,Linghu,X.,Li,P.,Wang,Y.,Li,Q.,Zhu, S.C., Jia, B., Huang, S., 2023. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871

    2023 arXiv

  35. [41]

    3d-llm: Injecting the 3d world into large language models

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C., 2023b. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems 36, 20482– 20494

  36. [42]

    Jia,B.,Chen,Y.,Yu,H.,Wang,Y.,Niu,X.,Liu,T.,Li,Q.,Huang,S.,

  37. [44]

    From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities

    Ishmam, M.F., Shovon, M.S.H., Mridha, M.F., Dey, N., 2024. From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities. Information Fusion 106, 102270

    2024

  38. [45]

    Ai2- thor: An interactive 3d environment for visual ai

    Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al., 2017. Ai2- thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474

    2017 arXiv

  39. [46]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Kudo, T., 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226

    2018 arXiv

  40. [47]

    Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.,

  41. [48]

    Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of naacL-HLT, Minneapolis, Minnesota

    Kenton, J.D.M.W.C., Toutanova, L.K., 2019. Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of naacL-HLT, Minneapolis, Minnesota. p. 2

    2019

  42. [49]

    M3dbench: Let’s instruct large models with multi-modal 3d prompts

    Li, M., Chen, X., Zhang, C., Chen, S., Zhu, H., Yin, F., Yu, G., Chen, T., 2023b. M3dbench: Let’s instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763

    arXiv

  43. [50]

    Li, Z., Yu, H., Yang, Z., Chen, T., Akhtar, N., 2023c. Ashapeformer: Semantics-guided object-level active shape encoding for 3d object detection via transformers, in: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 1012–1021

  44. [51]

    Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, pp

    Lin, C.Y., 2004. Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, pp. 74–81

    2004

  45. [52]

    Multi-modal situated reasoning in 3d scenes

    Linghu, X., Huang, J., Niu, X., Ma, X., Jia, B., Huang, S., 2024. Multi-modal situated reasoning in 3d scenes. arXiv preprint arXiv:2409.02389

    2024 arXiv

  46. [53]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, in: International conference on machine learning, PMLR

    Li, J., Li, D., Savarese, S., Hoi, S., 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, in: International conference on machine learning, PMLR. pp. 19730–19742

  47. [54]

    Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, pp

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021a. Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022

  48. [55]

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.,

  49. [56]

    Point-voxelcnnforefficient 3ddeeplearning

    Liu,Z.,Tang,H.,Lin,Y.,Han,S.,2019. Point-voxelcnnforefficient 3ddeeplearning. Advancesinneuralinformationprocessingsystems 32

    2019

  50. [57]

    Group-free 3d object detection via transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Liu, Z., Zhang, Z., Cao, Y., Hu, H., Tong, X., 2021b. Group-free 3d object detection via transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2949–2958

  51. [58]

    From screens to scenes: A survey of embodied ai in healthcare

    Liu,Y.,Cao,X.,Chen,T.,Jiang,Y.,You,J.,Wu,M.,Wang,X.,Feng, M., Jin, Y., Chen, J., 2025. From screens to scenes: A survey of embodied ai in healthcare. arXiv preprint arXiv:2501.07468

    2025 arXiv

  52. [59]

    Scalable 3d captioning with pretrained models

    Luo, T., Rockwell, C., Lee, H., Johnson, J., 2024b. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems 36

  53. [60]

    Sqa3d: Situated question answering in 3d scenes

    Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.C., Huang, S., 2022. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474

    2022 arXiv

  54. [61]

    11976– 11986

    A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976– 11986

  55. [62]

    Situational awareness matters in 3d vision language reasoning, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.13678– 13688

    Man, Y., Gui, L.Y., Wang, Y.X., 2024. Situational awareness matters in 3d vision language reasoning, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.13678– 13688

    2024

  56. [63]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, in: Findings of the Association for Computational Linguistics: ACL 2022, pp

    Masry, A., Do, X.L., Tan, J.Q., Joty, S., Hoque, E., 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, in: Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263–2279

    2022

  57. [64]

    Transformer- based vision-language alignment for robot navigation and question answering

    Luo, H., Guo, Z., Wu, Z., Teng, F., Li, T., 2024a. Transformer- based vision-language alignment for robot navigation and question answering. Information Fusion 108, 102351

  58. [65]

    Situated language understandingat25milesperhour,in:Proceedingsofthe15thAnnual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp

    Misu, T., Raux, A., Gupta, R., Lane, I., 2014. Situated language understandingat25milesperhour,in:Proceedingsofthe15thAnnual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 22–31

    2014

  59. [66]

    Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp

    Mo, W., Liu, Y., 2024. Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4261–4268

    2024

  60. [67]

    Openeqa: Embodied question answering in the era of foundation models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Majumdar,A.,Ajay,A.,Zhang,X.,Putta,P.,Yenamandra,S.,Henaff, M., Silwal, S., Mcvay, P., Maksymets, O., Arnaud, S., et al., 2024. Openeqa: Embodied question answering in the era of foundation models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  61. [68]

    Bleu: a method forautomaticevaluationofmachinetranslation,in:Proceedingsofthe 40thannualmeetingoftheAssociationforComputationalLinguistics, pp

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J., 2002. Bleu: a method forautomaticevaluationofmachinetranslation,in:Proceedingsofthe 40thannualmeetingoftheAssociationforComputationalLinguistics, pp. 311–318

    2002

  62. [69]

    Clip-guided vision-language pre- training for question answering in 3d scenes, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp

    Parelli, M., Delitzas, A., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., Hofmann, T., 2023. Clip-guided vision-language pre- training for question answering in 3d scenes, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp. 5607–5612

    2023

  63. [70]

    Wordnet: a lexical database for english

    Miller, G.A., 1995. Wordnet: a lexical database for english. Commu- nications of the ACM 38, 39–41

    1995

  64. [71]

    Glove: Global vec- tors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp

    Pennington, J., Socher, R., Manning, C.D., 2014. Glove: Global vec- tors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543

    2014

  65. [72]

    9277–9286

    Qi,C.R.,Litany,O.,He,K.,Guibas,L.J.,2019.Deephoughvotingfor 3d object detection in point clouds, in: proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9277–9286

    2019

  66. [73]

    Frozentransformersin language models are effective visual encoder layers

    Pang,Z.,Xie,Z.,Man,Y.,Wang,Y.X.,2023. Frozentransformersin language models are effective visual encoder layers. arXiv preprint arXiv:2310.12973

    2023 arXiv

  67. [74]

    Pointnet++: Deep hierarchicalfeaturelearningonpointsetsinametricspace

    Qi, C.R., Yi, L., Su, H., Guibas, L.J., 2017b. Pointnet++: Deep hierarchicalfeaturelearningonpointsetsinametricspace. Advances in neural information processing systems 30

  68. [75]

    Langsplat: 3d language gaussian splatting, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.20051– 20060

    Qin, M., Li, W., Zhou, J., Wang, H., Pfister, H., 2024. Langsplat: 3d language gaussian splatting, in: Proceedings of the IEEE/CVF ConferenceonComputerVisionandPatternRecognition,pp.20051– 20060

    2024

  69. [76]

    Openscene:3dsceneunderstandingwith open vocabularies, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., Funkhouser,T.,etal.,2023. Openscene:3dsceneunderstandingwith open vocabularies, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 815–824

    2023

  70. [77]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al., 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 9

    2019

  71. [78]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 1–67

    2020

  72. [79]

    Pointnet:Deeplearning on point sets for 3d classification and segmentation, in: Proceedings Z

    Qi,C.R.,Su,H.,Mo,K.,Guibas,L.J.,2017a. Pointnet:Deeplearning on point sets for 3d classification and segmentation, in: Proceedings Z. Li et al. :Preprint submitted to Elsevier Page 17 of 19 Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering of ...

  73. [80]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., et al., 2021. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238

    2021 arXiv

  74. [81]

    3d is here: Point cloud library (pcl), in: 2011 IEEE international conference on robotics and automation, IEEE

    Rusu, R.B., Cousins, S., 2011. 3d is here: Point cloud library (pcl), in: 2011 IEEE international conference on robotics and automation, IEEE. pp. 1–4

    2011

  75. [82]

    Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,Sastry,G.,Askell,A.,Mishkin,P.,Clark,J.,etal.,2021. Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR. pp. 8748– 8763

    2021

  76. [83]

    Mask3d: Mask transformer for 3d semantic instance segmentation, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE

    Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., Leibe, B., 2023. Mask3d: Mask transformer for 3d semantic instance segmentation, in: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 8216–8223

    2023

  77. [84]

    Evaluating zero-shot gpt-4v performance on 3d visual question answering benchmarks

    Singh, S., Pavlakos, G., Stamoulis, D., 2024. Evaluating zero-shot gpt-4v performance on 3d visual question answering benchmarks. arXiv preprint arXiv:2405.18831

    2024 arXiv

  78. [85]

    SQuAD: 100,000+ questions for machine comprehension of text, in: Su, J., Duh, K., Carreras, X

    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P., 2016. SQuAD: 100,000+ questions for machine comprehension of text, in: Su, J., Duh, K., Carreras, X. (Eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational ...

    2016 doi

  79. [86]

    Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.,

  80. [87]

    Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D.S., Maksymets, O., et al.,

  81. [88]

    Habitat: A platform for embodied ai research, in: Proceedings of the IEEE/CVF international conference on computer vision, pp

    Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al., 2019. Habitat: A platform for embodied ai research, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339–9347

    2019

  82. [89]

    Thai, A., Peng, S., Genova, K., Guibas, L., Funkhouser, T.,

  83. [90]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.,

  84. [91]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics

    Song, C.H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., Birch- field, S., 2024. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv preprint arXiv:2411.16537

    2024

  85. [92]

    Attention is all you need

    Vaswani, A., 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  86. [93]

    Cider: Consensus-based image description evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    Vedantam, R., Lawrence Zitnick, C., Parikh, D., 2015. Cider: Consensus-based image description evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575

    2015

  87. [94]

    Rio: 3d object instance re-localization in changing indoor environ- ments, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Wald, J., Avetisyan, A., Navab, N., Tombari, F., Nießner, M., 2019. Rio: 3d object instance re-localization in changing indoor environ- ments, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7658–7667

    2019

  88. [95]

    Self-instruct: Aligning language models with self-generated instructions

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., Hajishirzi, H., 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  89. [96]

    Space3d-bench: Spatial 3d question answering benchmark

    Szymanska, E., Dusmanu, M., Buurlage, J.W., Rad, M., Pollefeys, M., 2024. Space3d-bench: Spatial 3d question answering benchmark. arXiv preprint arXiv:2408.16662

    2024 arXiv

  90. [97]

    Medical vqa, in: Visual Question Answering: From Theory to Application

    Wu, Q., Wang, P., Wang, X., He, X., Zhu, W., 2022. Medical vqa, in: Visual Question Answering: From Theory to Application. Springer, pp. 165–176

    2022

  91. [98]

    arXiv preprint arXiv:2503.06271

    Splattalk: 3d vqa with gaussian splatting. arXiv preprint arXiv:2503.06271

    arXiv

  92. [99]

    Xia, F., Zamir, A.R., He, Z., Sax, A., Malik, J., Savarese, S.,

  93. [100]

    arXiv preprint arXiv:2302.13971

    Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    arXiv

  94. [101]

    Tyree, S., Tremblay, J., To, T., Cheng, J., Mosier, T., Smith, J., Birchfield, S., 2022. 6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark, in: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS...

    2022

  95. [103]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Yang,Z.,Li,L.,Lin,K.,Wang,J.,Lin,C.C.,Liu,Z.,Wang,L.,2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1

    2023 arXiv

  96. [104]

    3d question answering

    Ye, S., Chen, D., Han, S., Liao, J., 2021. 3d question answering. doi:10.1109/TVCG.2022.3225327

    2021

  97. [105]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Huang, X., Wang, Z., Sheng, L., Bai, L., et al., 2024. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems 36

    2024

  98. [106]

    Embodied question answeringinphotorealisticenvironmentswithpointcloudperception, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., Batra, D., 2019. Embodied question answeringinphotorealisticenvironmentswithpointcloudperception, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2019

  99. [107]

    Multi-target embodied question answering, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp

    Yu,L.,Chen,X.,Gkioxari,G.,Bansal,M.,Berg,T.L.,Batra,D.,2019. Multi-target embodied question answering, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp. 6309–6318

    2019

  100. [108]

    Building generalizable agents with a realistic and rich 3d environment

    Wu, Y., Wu, Y., Gkioxari, G., Tian, Y., 2018. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209

    2018 arXiv

  101. [109]

    Information fusion in visual question answering: A survey

    Zhang, D., Cao, R., Wu, S., 2019. Information fusion in visual question answering: A survey. Information Fusion 52, 268–280. Z. Li et al. :Preprint submitted to Elsevier Page 18 of 19 Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering

    2019

  102. [110]

    9068–9079

    Gibson env: Real-world perception for embodied agents, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9068–9079

  103. [111]

    Famma: A benchmark for financial domain multilingual multimodal question answering

    Xue, S., Chen, T., Zhou, F., Dai, Q., Chu, Z., Mei, H., 2024. Famma: A benchmark for financial domain multilingual multimodal question answering. arXiv preprint arXiv:2410.04526

    2024 arXiv

  104. [112]

    Yadav,K.,Ramrakhya,R.,Ramakrishnan,S.K.,Gervet,T.,Turner,J., Gokaslan, A., Maestre, N., Chang, A.X., Batra, D., Savva, M., et al.,

  105. [113]

    4927–4936

    Habitat-matterport3dsemanticsdataset,in:Proceedingsofthe IEEE/CVFConferenceonComputerVisionandPatternRecognition, pp. 4927–4936

  106. [114]

    2928–2937

    Zhao,L.,Cai,D.,Sheng,L.,Xu,D.,2021.3dvg-transformer:Relation modeling for visual grounding on point clouds, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2928–2937

    2021

  107. [115]

    IEEE Transactions on Visualization & Computer Graphics , 1–13

    Comprehensive visual question answering on point clouds through compositional scene manipulation. IEEE Transactions on Visualization & Computer Graphics , 1–13

  108. [116]

    Gimo:Gaze-informedhumanmotionpredictionincontext, in:EuropeanConferenceonComputerVision,Springer.pp.676–694

    Zheng,Y.,Yang,Y.,Mo,K.,Li,J.,Yu,T.,Liu,Y.,Liu,C.K.,Guibas, L.J.,2022. Gimo:Gaze-informedhumanmotionpredictionincontext, in:EuropeanConferenceonComputerVision,Springer.pp.676–694

    2022

  109. [117]

    Uni3d: Exploring unified 3d representation at scale

    Zhou, J., Wang, J., Ma, B., Liu, Y.S., Huang, T., Wang, X., 2023. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773

    2023 arXiv

  110. [118]

    3d- vista: Pre-trained transformer for 3d vision and text alignment, in: ProceedingsoftheIEEE/CVFInternationalConferenceonComputer Vision, pp

    Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., Li, Q., 2023. 3d- vista: Pre-trained transformer for 3d vision and text alignment, in: ProceedingsoftheIEEE/CVFInternationalConferenceonComputer Vision, pp. 2911–2921

    2023

  111. [119]

    Intervention and regulatory mechanism of multimodal fusion natural interactions on ar embodied cognition

    Yong, J., Wei, J., Lei, X., Wang, Y., Dang, J., Lu, W., 2025. Intervention and regulatory mechanism of multimodal fusion natural interactions on ar embodied cognition. Information Fusion 117, 102910

    2025

  112. [121]

    Empoweringlargelanguagemodelswith 3d situation awareness

    Yuan, Z., Peng, Y., Ren, J., Liao, Y., Han, Y., Feng, C.M., Zhao, H., Li,G.,Cui,S.,Li,Z.,2025. Empoweringlargelanguagemodelswith 3d situation awareness. arXiv preprint arXiv:2503.23024

    2025 arXiv

  113. [123]

    Opt: Open pre-trained transformer language models

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al., 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  114. [124]

    Vision-language pre-training with object contrastive learning for 3d scene understanding, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp

    Zhang, T., He, S., Dai, T., Wang, Z., Chen, B., Xia, S.T., 2024a. Vision-language pre-training with object contrastive learning for 3d scene understanding, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7296–7304

  115. [125]

    Spartun3d: Situated spatial understanding of 3d world in large language models

    Zhang, Y., Xu, Z., Shen, Y., Kordjamshidi, P., Huang, L., 2024b. Spartun3d: Situated spatial understanding of 3d world in large language models. arXiv preprint arXiv:2410.03878

    arXiv

  116. [126]

    His-gpt: Towards 3d human-in-scene multimodal understanding

    Zhao, J., Hou, R., Tian, Z., Chang, H., Shan, S., 2025. His-gpt: Towards 3d human-in-scene multimodal understanding. arXiv preprint arXiv:2503.12955

    2025 arXiv

  117. [128]

    Toward explainable 3d grounded visual question answering: A new benchmark and strong baseline

    Zhao, L., Cai, D., Zhang, J., Sheng, L., Xu, D., Zheng, R., Zhao, Y., Wang, L., Fan, X., 2022. Toward explainable 3d grounded visual question answering: A new benchmark and strong baseline. IEEE TransactionsonCircuitsandSystemsforVideoTechnology33,2935– 2949

    2022

  118. [132]

    Improved mlp point cloud processing with high-dimensional positional encoding, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp

    Zou, Y., Yu, H., Yang, Z., Li, Z., Akhtar, N., 2024. Improved mlp point cloud processing with high-dimensional positional encoding, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7891–7899. Z. Li et al. :Preprint submitted to Elsevier Page 19 of 19

    2024

  119. [2017]

    1746–1754

    Semantic scene completion from a single depth image, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1746–1754

  120. [2018]

    Embodied Question Answering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  121. [2021]

    Advances in neural information processing systems 34, 251–266

    Habitat2.0:Traininghomeassistantstorearrangetheirhabitat. Advances in neural information processing systems 34, 251–266

  122. [2022]

    arXiv preprint arXiv:2201.03546

    Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546

    arXiv

  123. [2023]

    arXiv preprint arXiv:2303.08774

    Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv

  124. [2025]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding, in: European Conference on Computer Vision, Springer. pp. 289–310

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.