Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ablating the 3D encoder leaves 3D VLM question-answering scores nearly unchanged.

desk verdict Useful diagnostic work on 3D scene-centric VLMs, but the headline encoder-ablation result is confounded by Q-Former text leakage, and the paper overgeneralizes from one model and one benchmark. read the letter →

arxiv 2506.05318 v2 pith:Z7JP7OXD submitted 2025-06-05 cs.CV

Haoyuan Li , Yanpeng Zhou , Yufei Gao , Tao Tang , Jianhua Han , Yujie Yuan , Dave Zhenyu Chen , Jiawang Bian
show 2 more authors
Hang Xu Xiaodan Liang
This is my paper
classification cs.CV
keywords 3Dvision-languagemodelsquestionansweringsceneencoderablationshortcutlearningQ-FormerScanQA3D-RDQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that 3D scene-centric vision-language models, which take a whole 3D scene as input, achieve their question-answering scores without genuinely using the 3D encoder. In systematic ablations of LL3DA on ScanQA, randomly reinitializing the encoder weights or zeroing the encoder outputs leaves BLEU-4, CIDEr, and ROUGE essentially unchanged, while removing all 3D input collapses BLEU-4 to 0.00. The authors conclude that these models lean on scene-agnostic Q-Former queries, question text, and memorized frequent answers, making the 3D encoder largely dispensable. They also find that the pretrain stage adds little and that data scaling flattens on larger datasets. To test and break this shortcut, they build a multiple-choice version of ScanQA and a 3D Relevance Discrimination QA dataset; on that dataset, removing the 3D input drops accuracy to 0%, showing the encoder can matter when shortcuts are removed.

What carries the argument

The load-bearing instrument is the ablation of the 3D encoder pathway in LL3DA: separately removing the encoder's pretrained weights, zeroing the encoder outputs, and removing the 3D input while keeping the Q-Former's 32 latent queries. This isolates whether the Q-Former queries, which are initialized without scene content, carry the task performance. The complementary instrument is the 3D Relevance Discrimination QA dataset, constructed by pairing each multiple-choice question with a poisoned copy whose 3D tokens come from a different scene, so that a model must consult the 3D tokens to pick the right answer. Together they separate reliance on text and answer frequency from genuine use of geometry.

What would settle it

Run the same three-way ablation on a second scene-centric model, for example Grounded 3D-LLM or LSceneLLM, under the same one-epoch schedule: if removing encoder weights or zeroing encoder outputs changes ScanQA BLEU-4 by more than a few points, the central claim fails for that architecture. Alternatively, train LL3DA with several epochs of pretraining and check whether encoder ablation then produces a large performance drop.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the understanding of 3D scene information in scene-centric 3D VLMs rests on the scene-agnostic latent queries of the Q-Former rather than on features extracted by the 3D scene encoder. Ablating the encoder's pretrained weights or all encoder outputs leaves ScanQA BLEU-4 at 14.31 and 13.68 versus 13.82 with the full model, while removing the 3D input entirely drops BLEU-4 to 0.00. The same pattern appears in the training stages: one-epoch pretraining gives no significant benefit over starting directly from SFT, and scaling datasets from 145k to 661k samples produces no consistent gain. The paper traces the mechanism to answer-distribution overfitting, with models emitting frequent answers far more often than the ground truth does, and verifies the diagnosis with a poisoned-pair dataset where models that truly use 3D tokens must distinguish original from altered scenes.

Load-bearing premise

The central claim rests on treating LL3DA with the Q-Former, ScanQA, and a one-epoch pretrain-then-SFT schedule as representative of 3D scene-centric VLMs; if another architecture or a longer training schedule makes encoder ablations matter, the conclusion weakens.

Editorial extensions

If this is right

  • If current scene-centric 3D VLMs ignore the encoder, published ScanQA results overstate genuine 3D understanding and should not be read as evidence of spatial reasoning.
  • The pretrain-then-SFT recipe transferred from 2D VLMs does not automatically help 3D scene-centric models under a one-epoch schedule.
  • Cross-task data scaling saturates beyond roughly 135k samples, so simply adding more 3D QA data is unlikely to yield further gains for these models.
  • Multiple-choice evaluation on ScanQA-Choice exposes that models rely on option structure and question text rather than on 3D tokens.
  • The 3D-RDQA dataset makes the 3D encoder and pretraining influence the outcome, indicating that shortcut-breaking data can push models toward genuine 3D scene understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same three-way ablation could be applied to 2D image-based and 3D object-centric VLMs; the authors hint the problem may be shared across 3D VLM families, but they do not demonstrate it, so that generalization remains untested.
  • Editorial inference: the score gap between a full model and a no-3D-input variant is a simple, reusable diagnostic for encoder reliance that could be reported for every new 3D VLM.
  • Editorial inference: the central claim may be schedule-dependent, since the paper trains only one epoch of pretraining and SFT; longer pretraining could give the encoder more opportunity to become load-bearing.
  • Editorial inference: because the paper sketches relevance-discrimination pairs for dense captioning and visual grounding, the same data-centric test could grow into a full benchmark suite for 3D spatial understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies why 3D scene-centric VLMs underperform compared with object-centric and image-based 3D VLMs, using LL3DA as a representative baseline and ScanQA as the primary benchmark. Through ablations, the authors report three observations: (i) removing or randomly initializing the 3D scene encoder's weights or outputs leaves ScanQA scores nearly unchanged, while removing Q-Former outputs collapses performance; (ii) one-epoch pretraining before SFT gives little benefit; and (iii) data scaling helps on small datasets but saturates beyond roughly 135k samples. They then test semantic content, question format, and answer distributions, introducing ScanQA-Choice and the 3D Relevance Discrimination QA (3D-RDQA) dataset, and conclude that current 3D scene-centric VLMs over-rely on text cues and answer frequencies rather than on 3D geometry.

Significance. If the central observation holds, the paper provides a timely and useful diagnosis for the 3D VLM community: it would imply that many reported gains on ScanQA reflect text-side shortcuts rather than genuine 3D understanding, and that pretraining recipes transferred from 2D VLMs may need rethinking. The paper's strengths are its extensive, internally consistent ablations (Tables 1-4, 6-9), its release of ScanQA-Choice and 3D-RDQA, and its explicit discussion of limitations. The proposed 3D-RDQA protocol is a constructive way to force 3D tokens to matter, and the authors honestly acknowledge in Appendix J.2 that the 0% accuracy without 3D input reflects the constructed answer distribution. The significance is bounded, however, by the single-architecture, single-benchmark evidence base and by a confound in the headline encoder-ablation experiment.

major comments (4)
  1. [Section 3.1, Table 1; Section 4.2, Table 6] The headline claim that the 3D encoder's weights and outputs are dispensable on ScanQA is confounded by the Q-Former. In Table 1, the zeroing or randomization is applied to encoder weights or outputs while the Q-Former remains in the loop, and the Q-Former may route question text to the LLM. The paper itself acknowledges this risk in the Table 6 caption, where it says it switches to two-layer MLPs 'to avoid Q-Former directly learning the text embedding of the question.' However, the MLP experiments are run on ScanQA-Choice and 3D-RDQA (Tables 6, 9), not on the open-ended ScanQA setting used in Table 1. The invariance of BLEU-4, CIDEr, and ROUGE under encoder ablation could therefore reflect text leakage through the Q-Former rather than the model ignoring 3D features. Please de-confound the headline result by repeating the Table 1 ablations on open-ended ScanQA with an MLP projector or by ablating the Q-Former's text input path.
  2. [Section 3.2, Table 2; Section B] Observation 2 (pretraining is less effective) and Observation 3 (data scaling saturates) are established with only one epoch of pretraining and one epoch of SFT. Section B states that one epoch 'can yield comparable performance' to LL3DA's 32-epoch Q-Former training, but comparable final performance does not mean that the effect of pretraining has been given a fair chance to appear; similarly, saturation in Tables 3 and 4 may be a symptom of undertraining rather than a property of the data. Please report learning curves or ablations with longer schedules on at least the pretraining and scaling experiments, or explicitly reword the conclusions to apply to the one-epoch setting.
  3. [Section 3; Appendix J] The paper's title, abstract, and conclusion generalize to '3D scene-centric VLMs' and even 'all 3D VLMs', but the experiments use a single architecture (LL3DA with Q-Former) and a single benchmark (ScanQA, with ScanQA-Choice and 3D-RDQA derived from it). The evidence for other architectures is qualitative (Appendix J.1 discusses 3D-LLM, Grounded 3D-LLM, and LSceneLLM by citing their reported numbers), and no second model is trained under the same protocol. Please either add a second scene-centric baseline to the core ablations or weaken the general claims to the tested model family.
  4. [Section 4.4, Table 9; Appendix J.2] The 3D-RDQA verification is a self-constructed diagnostic and is not independent evidence for the ScanQA conclusion. In Table 9, the 0% accuracy of the no-3D-input model arises, as the authors explain in Appendix J.2, because the training set's Penalty QA items make 'E' the most frequent answer and the test set contains only Regular QA items. This shows the designed shortcut exists in 3D-RDQA, but it does not validate the transfer of the encoder-dispensability conclusion back to open-ended ScanQA. Please present 3D-RDQA as an illustrative stress test and not as independent confirmation of Observation 1.
minor comments (5)
  1. [Tables 1-4; Section 3.2 heading] Several table captions write 'BLUE-4' instead of 'BLEU-4', and the Section 3.2 heading reads 'Dose pre-training stage matter?' instead of 'Does pre-training stage matter?'.
  2. [Table 5 and Appendix C] The scene-description experiments use a sampled subset (ScanQA*) and only the final 100 scenes of the train split as a test set; the paper reports no variance or repeated splits, so small differences (e.g., 5.18 vs. 5.40) should be interpreted cautiously.
  3. [Figure 3 and Appendix H] The answer-frequency figures lack labeled axes, units, and a clear explanation of the black dashed box; adding these would make the overfitting point easier to verify.
  4. [Section 2.2 and Table 1] The paper says LL3DA trains only the Q-Former, but the baseline rows in Table 1 include different LLMs (Opt-1.3B vs. Qwen2-1.5B) and different training protocols; please clarify which parameters are frozen in each baseline so the reader can compare like with like.
  5. [Appendix J.4, Eqs. (1)-(3)] The normalization for bounding-box sizes uses scene coordinate ranges for all dimensions; please clarify whether this is intentional for width/height/length and whether Signed Normalization can produce negative values for w, h, or l.

Circularity Check

1 steps flagged · score 2.0 of 10

Central claim is supported by external-benchmark ablations; only the 3D-RDQA 'verification' is partly entailed by its own construction, and the paper discloses this.

  1. other [Section 4.4 'Summary and Verification' and Appendix J.2 'Performance analysis on 3D-RDQA dataset']
    "Our 3D-RDQA pair construction involves a Penalty QA item for each question, where the answer is consistently "E," contrasting with the even distribution of Regular QA answers across A, B, C, and D. This design leads to a much higher occurrence of "E" in the training data. As the test set lacks these Penalty QA items, the text-dependent 3D VLM (without the 3D encoder) defaults to the most frequent trained answer, "E," leading to a 0% accuracy."

    The 0% accuracy for the no-3D-input condition in Table 9 is entailed by the dataset construction (Penalty QA always answered "E" and absent at test), not discovered empirically. Citing this result as validation of the findings makes the verification partly self-confirming: the dataset was built so that a text-only model must fail. The central claim nevertheless rests on independent ablations on the external ScanQA benchmark (Table 1), so this is a minor, disclosed circularity in the validation step rather than a load-bearing derivation.

full rationale

The paper's three observations are direct ablations on the externally defined ScanQA benchmark, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain; LL3DA, 3D-LLM, and related baselines are cited as prior work, not as uniqueness constraints. The possible Q-Former text-leakage confound in Table 1 is a validity concern that the authors themselves disclose in the Table 6 caption ('to avoid Q-Former directly learning the text embedding of the question'), and it is not a circularity. The only construction-dependent element is the 3D-RDQA verification, where the no-encoder 0% is produced by the E-answer penalty design; the paper states this mechanism openly in Appendix J.2. Because the central conclusions are established independently on ScanQA, the overall circularity is low (score 2).

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claims rest on a small set of modeling choices: one representative architecture, one benchmark, and a specific construction of poisoned data. The free parameters are minimal, but the axiomatic assumptions about representativeness and the validity of the poisoning strategy are load-bearing. The dataset itself is an invented artifact, albeit a publicly available one.

free parameters (1)
  • grid scaling g in Eq. (1) = 255
    Chosen normalization constant for 3D-VG bounding box coordinates in Appendix J.4, used to scale normalized coordinates. It is a hand-set value, not central to the main claim about encoder reliance.
assumptions (4)
  • domain assumption LL3DA is representative of 3D scene-centric VLMs
    All main experiments use LL3DA as the baseline (Section 3). The paper generalizes its findings to all 3D scene-centric VLMs without testing other architectures.
  • domain assumption ScanQA is a representative benchmark for 3D-QA
    The paper selects ScanQA as the primary benchmark and uses its results to draw general conclusions about data scaling and pretraining for 3D VLMs (Section 3).
  • ad hoc to paper One epoch of pretraining and one epoch of SFT are sufficient to assess pretraining effectiveness
    The pretraining ablation in Section 3.2 trains only one epoch each. The paper does not show that longer pretraining would remain ineffective, despite claiming pretraining is overall less significant.
  • domain assumption Swapping 3D tokens from a different scene removes relevant 3D information
    3D-RDQA construction in Appendix I assumes that replacing a scene's 3D tokens with tokens from another scene makes the pair 'false' or poisoned. This is plausible but not validated against other ways of breaking 3D information.
invented entities (1)
  • 3D-RDQA dataset independent evidence
    purpose: A paired QA dataset with poisoned 3D tokens, designed to break shortcut learning and encourage the model to use 3D scene information.
    The dataset is publicly released via GitHub, so other researchers can run models on it and test the same diagnostic. It provides a falsifiable handle, though its ability to improve genuine 3D understanding on standard benchmarks is not yet demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs." pith.science (2026). https://pith.science/paper/Z7JP7OXD

@misc{pith2026250605318,
  author       = {Pith},
  title        = {Pith review of: Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7JP7OXD}},
  note         = {Machine review of arXiv:2506.05318}
}
read the original abstract

Remarkable progress in 2D Vision-Language Models (VLMs) has spurred interest in extending them to 3D settings for tasks like 3D Question Answering, Dense Captioning, and Visual Grounding. Unlike 2D VLMs that typically process images through an image encoder, 3D scenes, with their intricate spatial structures, allow for diverse model architectures. Based on their encoder design, this paper categorizes recent 3D VLMs into 3D object-centric, 2D image-based, and 3D scene-centric approaches. Despite the architectural similarity of 3D scene-centric VLMs to their 2D counterparts, they have exhibited comparatively lower performance compared with the latest 3D object-centric and 2D image-based approaches. To understand this gap, we conduct an in-depth analysis, revealing that 3D scene-centric VLMs show limited reliance on the 3D scene encoder, and the pre-train stage appears less effective than in 2D VLMs. Furthermore, we observe that data scaling benefits are less pronounced on larger datasets. Our investigation suggests that while these models possess cross-modal alignment capabilities, they tend to over-rely on linguistic cues and overfit to frequent answer distributions, thereby diminishing the effective utilization of the 3D encoder. To address these limitations and encourage genuine 3D scene understanding, we introduce a novel 3D Relevance Discrimination QA dataset designed to disrupt shortcut learning and improve 3D understanding. Our findings highlight the need for advanced evaluation and improved strategies for better 3D understanding in 3D VLMs.

Figures

Figures reproduced from arXiv: 2506.05318 by the authors.

Figure 1
Figure 1. Left: 3D VLM (Vision Language Model) with encoder leverages 3D Encoder to "see" scenes for question answering. Middle: 3D VLM without Encoder direct outputs answer. Right: 3D VLMs with and without an encoder achieve similar performance, but why? Abstract Remarkable progress in 2D Vision-Language Models (VLMs) has spurred interest in extending them to 3D settings for tasks like 3D Question Answering, Dense Captioning… view at source ↗
Figure 2
Figure 2. Visualization of different 3D VLM patterns. Similar to 2D VLM, 3D VLM also requires an encoder to extract features that serve as 3D tokens for the cross-modal input. Variations in the model design primarily stem from the choice of encoder: (a) utilizing a 3D object encoder necessitates initial object detection and subsequent relation modeling, (b) employing a 2D image encoder requires rendering the 3D scene into a s… view at source ↗
Figure 3
Figure 3. Analysis of generated answer frequency. The top 20 generated answers under various training settings show that test data inclusion did not improve fitting to frequent answers and might generalize them to other questions. Having excluded factors of semantic information and question format, we observed that the model excessively relies on textual questions to model the relationship between questions and answers. This … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visualization of token distribution with different cross-modal input. We further visualize the token distribution before and after MLP projector to intuitively express the impact of pre-train stage [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Example visualization of ScanQA and ScanQA-Choice collection. Based on the ground truth answer for each question in ScanQA, we sampled similar options from the ScanQA answer pool to construct ScanQA-Choice. Moreover, as shown in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Further analysis of generated answer frequency. The top 50 generated answers under various training settings [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Model Comparison: (a) A "dumb" model ignores 3D tokens, relying only on text, (b) A "smart" model understands 3D tokens and their relation to text. 3D tokens with "bomb" denotes the poisoned 3D tokens. J.2 Performance analysis on 3D-RDQA dataset As shown in [PITH_FULL…
Figure 8
Figure 8. Figure 8: Example visualization of 3D-RDQA pair collection. Utilizing our constructed ScanQA￾Choice dataset, we generate a 3D-RDQA pair by modifying 3D tokens and introducing a novel choice option. LLM 3D input Pre-train SFT Strategy of mixture Accuracy ↑ Qwen2-1.5B ✓ batch conc…
Figure 9
Figure 9. Figure 9: Example visualization of 3D-RDDC (3D Relevance Discrimination Dense Captioning) pair collection. Unlike in 3D-QA, 3D VLMs on 3D-DC tasks might over-rely on the provided 3D position information rather than the question itself. The core idea of 3D-RDQA is to construct co…
Figure 10
Figure 10. Figure 10: Example visualization of 3D-RDVG (3D Relevance Discrimination Visual Grounding) pair collection. 3D VLMs on 3D-VG tasks may perform similar to 3D-QA tasks due to the description provided in question, which resulting in similar way to design relevance discrimination da…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAPruner: Conceptual-Adjacent Scene Graph Pruner for Enhancing 3D Spatial Reasoning of Large Language Models

    cs.CL 2026-04 accept novelty 6.0 of 10

    Task-aware scene-graph pruning that mixes fuzzy semantic relevance with proximity measurably improves LLM 3D spatial reasoning over proximity-only KNN.

Reference graph

Works this paper leans on

55 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [36]

    More text, less point: Towards 3d data-efficient point-language understanding

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Jinfeng Xu, Yixue Hao, Long Hu, and Min Chen. More text, less point: Towards 3d data-efficient point-language understanding. InProceedings of the AAAI Conference on Artificial Intelligence, pages 7284–7292, 2025. 3

    work page 2025

  2. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 422–440. Springer, 2020. 2, 13

    work page 2020

  3. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022. 2, 13

    work page 2022

  4. [3]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean conference on computer vision, pages 202–221. Springer, 2020. 2, 13

    work page 2020

  5. [4]

    Language conditioned spatial relation reasoning for 3d object grounding.Advances in neural information processing systems, 35:20522–20535, 2022

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding.Advances in neural information processing systems, 35:20522–20535, 2022. 3

    work page 2022

  6. [5]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11124–11133, 2023. 3

    work page 2023

  7. [6]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428–26438, 2024. 2, 3, 4, 5, 13

    work page 2024

  8. [7]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11):7331–7347, 2024

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11):7331–7347, 2024. 2, 3, 13

    work page 2024

Show all 55 references
  1. [8]

    Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024. 2, 3, 4, 17

    2024 arXiv

  2. [9]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203, 2021. 2, 5

    2021

  3. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  4. [11]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 2, 13

    2017

  5. [12]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.arXiv preprint arXiv:2401.16420, 2024

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.arXiv preprint arXiv:2401.1642...

    2024 arXiv

  6. [13]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following.arXiv preprint arXiv:2309.00615, 2023

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following.arXiv preprint arXi...

    2023 arXiv

  7. [14]

    3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023. 2, 3, 4, 6, 7, 13, 14, 15, 16, 17

    2023

  8. [15]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.CoRR, 2023. 2, 4

    2023

  9. [16]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 2, 3, 4, 5, 6, 19

    2023 arXiv

  10. [17]

    Clip2point: Transfer clip to point cloud classification with image-depth pre-training

    Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22157–...

    2023

  11. [18]

    Frozen clip model is efficient point cloud backbone.arXiv preprint arXiv:2212.04098, 1(6), 2022

    Xiaoshui Huang, Sheng Li, Wentao Qu, Tong He, Yifan Zuo, and Wanli Ouyang. Frozen clip model is efficient point cloud backbone.arXiv preprint arXiv:2212.04098, 1(6), 2022. 3 10

    2022 arXiv

  12. [19]

    Pointgroup: Dual-set point grouping for 3d instance segmentation

    Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pages 4867–4876, 2020. 3

    2020

  13. [20]

    Unigs: Unified language-image-3d pretraining with gaussian splatting.arXiv preprint arXiv:2502.17860, 2025

    Haoyuan Li, Yanpeng Zhou, Tao Tang, Jifei Song, Yihan Zeng, Michael Kampffmeyer, Hang Xu, and Xiaodan Liang. Unigs: Unified language-image-3d pretraining with gaussian splatting.arXiv preprint arXiv:2502.17860, 2025. 3

    2025 arXiv

  14. [21]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023. 2, 4, 13

    2023

  15. [22]

    3dmit: 3d multi-modal instruction tuning for scene understanding

    Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu, and Rong Wei. 3dmit: 3d multi-modal instruction tuning for scene understanding. In2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pages 1–5. IEEE, 2024. 2, 3, 4

    2024

  16. [23]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004. 13

    2004

  17. [24]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  18. [25]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 2, 5, 13

    2023

  19. [26]

    Openshape: Scaling up 3d shape representation towards open-world understanding.Advances in neural information processing systems, 36:44860–44879, 2023

    Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding.Advances in neural information processing systems, 36:44860–44879, 2023. 3

    2023

  20. [27]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 13

    2017 arXiv

  21. [28]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022. 2, 5

    2022 arXiv

  22. [29]

    Image caption generation using vision transformer and gpt architecture

    Swapneel Mishra, Saumya Seth, Shrishti Jain, Vasudev Pant, Jolly Parikh, Rachna Jain, and Sardar MN Islam. Image caption generation using vision transformer and gpt architecture. In2024 2nd International Conference on Advancement in Computation & Computer Technologies (InCACCT...

  23. [30]

    An end-to-end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. InProceedings of the IEEE/CVF international conference on computer vision, pages 2906–2917, 2021. 3

    2021

  24. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 13

    2002

  25. [32]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017. 3

    2017

  26. [33]

    Deep hough voting for 3d object detection in point clouds

    Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. Inproceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019. 3

    2019

  27. [34]

    Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. InProceedings of the 32nd ACM International Conference on Multimedia, pages 6617–6626, 2024. 3

    2024

  28. [35]

    Exploring the potential of encoder-free architectures in 3d lmms.arXiv preprint arXiv:2502.09620, 2025

    Yiwen Tang, Zoey Guo, Zhuhao Wang, Ray Zhang, Qizhi Chen, Junli Liu, Delin Qu, Zhigang Wang, Dong Wang, Xuelong Li, et al. Exploring the potential of encoder-free architectures in 3d lmms.arXiv preprint arXiv:2502.09620, 2025. 3

    2025

  29. [37]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  30. [38]

    Consensus-based image description evaluation

    R Vedantam, C Lawrence Zitnick, and D Parikh. Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575. 13

  31. [39]

    Rio: 3d object instance re-localization in changing indoor environments

    Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re-localization in changing indoor environments. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7658–7667, 2019. 5 11

    2019

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  33. [41]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023. 2, 3, 4

    2023 arXiv

  34. [42]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.Advances in Neural Information Processing Systems, 37:69925–69975,

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.Advances in Neural Information Processing Sys...

  35. [43]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding

    Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. InProceedings of the IEEE/CVF conference on co...

    2023

  36. [44]

    Ulip-2: Towards scalable multimodal pre-training for 3d understanding

    Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. InProceedings of the IEEE/CVF Conference on Computer Vision ...

    2024

  37. [45]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  38. [46]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19313–19322, 2022. 3

    2022

  39. [47]

    Clip2: Contrastive language-image-point pretraining from real-world point cloud data

    Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. InProceedings of the IEEE/CVF conference on computer vi...

    2023

  40. [48]

    Pointclip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022. 3

    2022

  41. [49]

    Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders

    Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21769–21780, 2023. 3

    2023

  42. [50]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225– 15236, 2023. 5

    2023

  43. [51]

    Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences

    Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, and Chuang Gan. Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences. arXiv preprint arXiv:2412.01292, 2024. 2, 3, 4, 17, 19

    2024 arXiv

  44. [52]

    Uni3d: Exploring unified 3d representation at scale.arXiv preprint arXiv:2310.06773, 2023

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale.arXiv preprint arXiv:2310.06773, 2023. 3

    2023 arXiv

  45. [53]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024. 2, 3, 4, 6

    2024 arXiv

  46. [54]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

  47. [55]

    it is to the

    Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. InProceedings of the IEEE/CVF international conference on computer vision, pages 2639–2650, 2023. 3...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.