Pith. sign in

REVIEW 2 major objections 9 minor 62 references

Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

T0 review · 2 major / 9 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 3D vision-language model that grounds language through a mask decoder over shared dense features, beating prior 3D LMMs on ScanRefer grounding and ScanNet200 segmentation.

desk verdict Solid 3D LMM with a plausible architecture, but the central claim that dense decoding is the cause of the gains rests on a shaky 50-scene ablation; send it to review with a request for a matched baseline. read the letter →

arxiv 2608.02980 v1 pith:HUVIGOD3 submitted 2026-08-04 cs.CV

Lucy Lin , Ayush Jain , Yifan Liu , Katerina Fragkiadaki This is my paper · ORCID
classification cs.CV
keywords 3Dvision-languagemodelreferentialgroundinginstancesegmentationrotarypositionalembeddingsmaskdecodingmulti-viewRGB-DScanReferNet200
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the main reason 3D vision-language models underperform specialist perception systems is not their scene representation but their decoding interface: passing grounding through text tokens, proposal selection, or a single special token chokes off dense geometric information. Qwen-3D instead lets the language model and a query-based mask decoder share the full contextualized visual representation, so object queries attend directly to scene geometry and language context. Trained jointly on 2D and 3D data, it reports state-of-the-art results among 3D LMMs on ScanRefer (62.8 Acc@25 for the 7B model) and ScanNet200 instance segmentation (25.3 mAP for the 3B model), and it beats a specialist baseline on out-of-distribution ScanNet++ scenes. If the claim holds, geometry-aware decoding, not just geometry-aware attention, is what closes the gap between generalist 3D LMMs and specialist 3D perception.

What carries the argument

The load-bearing mechanism is the grounding decoder's direct access to the full contextualized token set. Multi-view RGB-D frames are lifted to a world-aligned point-cloud token set via unprojection and 5-cm voxel mean-pooling; the Qwen2.5-VL vision-language attention layers are adapted with 3D RoPE (four positional components: temporal plus x, y, z with uniform frequency ranges) and full (non-causal) attention over visual tokens; and a Mask2Former-style query decoder predicts masks by dot-product with the shared visual tokens and text-span grounding scores by dot-product with the language tokens. The decoder's alternating cross-attention to visual and language tokens is what replaces the bottleneck interfaces the paper criticizes.

What would settle it

Feed Qwen-3D a multi-view sequence in which an object visibly moves between views (or perturb camera poses beyond the noise levels tested in the appendix) and measure grounding accuracy at IoU@0.25: the claim that dense shared features are the key bottleneck would be undercut if accuracy collapses to the level of a 2D-per-frame baseline despite intact language reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 3D large multimodal model should ground language in geometry by sharing dense visual representations with a mask-based segmentation decoder, rather than by autoregressing coordinates as text, selecting among object proposals, or compressing grounding through a single dedicated token (e.g., <REF>). Qwen-3D realizes this by unprojecting multi-view RGB-D features into a shared world coordinate system, voxel-pooling them into geometry-aligned tokens, applying 3D Rotary Positional Embeddings so attention operates in 3D space, and routing the contextualized vision-language tokens into a Mask2Former-style decoder whose object queries cross-attend to both visual tokens and language tokens. The authors report that this architecture surpasses prior 3D LMMs on 3D referential grounding, matches or exceeds expert models on ScanRefer, leads 3D LMMs on ScanNet200 instance segmentation, and generalizes better out of distribution than specialist baselines on ScanNet++, while retaining strong 2D vision-language performance.

Load-bearing premise

The model assumes a static world and trustworthy external depth maps and camera poses: if the scene moves or the geometry estimates are wrong, the world-aligned tokens and everything grounded through them are invalid.

Editorial extensions

If this is right

  • 3D LMMs can approach specialist-level grounding and segmentation in-domain instead of trailing by large margins.
  • Out-of-distribution generalization can exceed that of specialist models when grounding is decoded from VLM features rather than from external detector proposals.
  • Long multi-view video streams can be compressed into a single world-aligned token set, making ~90-frame scene inference feasible at roughly two seconds per scene.
  • Joint 2D/3D training preserves the backbone's 2D grounding and QA ability, so 3D capability can be added without catastrophic forgetting.
  • Geometry-aware representation alone is not sufficient; the decoding interface must also be geometry-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same dense-sharing design could extend to dynamic 4D scenes if temporal state is integrated into the world-aligned token set, a direction the paper flags as an open problem.
  • The decoder's cross-attention to language tokens suggests a scalable route to multi-object grounding and compositional instructions, since each object query can bind to any text span rather than a single special token.
  • A testable consequence is that any autoregressive or bottlenecked interface (text-box decoding, <REF> tokens, proposal selection) should consistently underperform dense-sharing on fine-grained instances, not only on the benchmarks reported here.
  • If the bottleneck claim is right, improving the representation further (e.g., better depth/pose or learned geometry) should compound with the decoder rather than be limited by it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 9 minor

Summary. The paper presents Qwen-3D, a 3D vision-language model that extends Qwen2.5-VL by unprojecting multi-view RGB-D features into a shared world coordinate system, voxel-pooling them, and applying 3D Rotary positional embeddings and full attention in the vision-language layers. Grounding and instance segmentation are produced by a Mask2Former-style query decoder that consumes the full contextualized visual and text token sets, while question answering uses the standard text head. The authors report state-of-the-art results among 3D LMMs on ScanRefer, a large OOD margin on ScanNet++, competitive ScanNet200 instance segmentation, and preservation of 2D capabilities, with ablations isolating the decoder interface, positional encoding, attention mask, and VLM fine-tuning.

Significance. The central architectural proposal -- replacing text-space, proposal-selection, or special-token grounding interfaces with a dense shared representation feeding a mask decoder -- is well motivated and, if confirmed, would be a useful design principle for 3D LMMs. The paper's strengths are the breadth of evaluation (3D grounding, instance segmentation, 3D VQA, 2D retention, OOD generalization, noise robustness), the clear ablations for attention masking and VLM fine-tuning, and the stated commitment to release code and checkpoints. The main weakness is that the single ablation that isolates the central interface claim is small and statistically uncontrolled, and the 3D positional-embedding ablation is too weak to support the associated contribution.

major comments (2)
  1. [Section 4.6, Table 5a] The central claim that the full-token interface, not the <REF> bottleneck, causes the grounding improvement rests entirely on this ablation, but it is uncontrolled and underpowered. All numbers are from 50 randomly sampled ScanRefer scenes with no error bars or multiple seeds; ScanRefer utterances vary widely in difficulty, so a 13.8-point gap on this subset may be within sampling noise. The Qwen-3D<REF> variant scores 39.7, below the published full-validation Acc@50 of Grounded-3D-LLM (44.0) and LLaVA-3D (42.7), even though the full Qwen-3D model at 3B scale beats these 7B models; this makes a weak implementation of the <REF> path a plausible alternative explanation. Please specify how the referent token is produced, whether the mask decoder in that variant attends only to that token or also to visual tokens, what loss weights and training schedule are used, and report repeated runs or full-validation results for a matched <REF> control. The published <REF> baselines in the same table are not matched for data mixture, LoRA rank, or iteration count, so they cannot substitute for a matched control.
  2. [Section 4.6, Table 5b] The claimed benefit of 3D Rotary PE over 2D MRoPE is 53.5 vs 53.2, a 0.3-point difference on the same 50-scene subset; without error bars or multiple seeds this is indistinguishable from noise, especially since the 'vanilla 3D' variant is 3.6 points worse than 2D. This is too weak to support the contribution that geometry-aware attention 'enables cross-view spatial reasoning' (Sections 1 and 3.2). The authors should report full-validation numbers, repeated seeds, or a paired statistical test, and should temper the claim if the effect does not hold.
minor comments (9)
  1. [Section 3.4] The probability p for lifting 2D data to 3D is never specified; please state its value or how it is chosen.
  2. [Equation (1)] The loss weights alpha_mask, alpha_textground, and alpha_gen are not reported anywhere; please give their values or a reference to a configuration file.
  3. [Table 5 caption] The caption should state the metric (Top-1 Acc@0.5), the number of scenes, and that all entries are single-run results; the text currently gives this information only indirectly.
  4. [Section 2] The phrase 'parameter-free manner' for voxel-based compression is misleading because the voxel size is a tuned hyperparameter; please say 'no learned parameters' instead.
  5. [Figure 2 and Section 2] The text refers to decoder designs as (a)-(d), while the figure labels them (i)-(iv); please unify the labels.
  6. [Section 4.1, Table 1] The sentence claiming Qwen-3D is 'outperforming all other methods' on ScanRefer should be qualified, since UniVLG's 63.5 Acc@25 is higher than both Qwen-3D variants (62.2 and 62.8).
  7. [Section 5] The static-world and external depth/pose assumptions are acknowledged, but Appendix 8.1 only injects Gaussian noise; please state explicitly that systematic pose drift or dynamic scenes are outside the scope of the robustness claims.
  8. [Table 2] The ScanNet++ comparison should document the exact protocol for UniVLG and Video-3D-LLM (prompts, post-processing, official checkpoints, and how masks are converted to boxes) so that the large OOD margin can be reproduced.
  9. [Appendix 8.7] The frequency ranges used for the aligned 3D RoPE are not specified; at minimum report the ranges or give the formula used to initialize them.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: all claimed results are held-out benchmark numbers from a trained system, not fitted constants; the central interface claim rests on an internal (though underpowered) ablation rather than on self-citation or definitional equivalence.

full rationale

The paper's derivation chain is empirical and externally benchmarked. All headline numbers (ScanRefer 62.8 Acc@25 for 7B, ScanNet200 25.3 mAP for 3B, OOD ScanNet++ 55.7 Acc@25) are measured on held-out validation or test splits after supervised training on disjoint train splits; no headline number is obtained by fitting a constant to the benchmark it is then used to predict. The central architectural claim—that sharing full contextualized vision-text features with a mask decoder outperforms a <REF>-token or proposal-selection bottleneck—is supported by Table 5a, an internal ablation of the authors' own model on 50 ScanRefer scenes. That ablation is uncontrolled (no matched <REF> implementation details, no error bars) and therefore is weak evidence, but methodological weakness is a correctness risk, not circularity: the comparison is not a definitional identity, and the published <REF>-token baselines (Grounded-3D-LLM 44.0, LLaVA-3D 42.7) are independent external numbers above the paper's own <REF> variant (39.7), so the result is not forced by construction. Self-citations (UniVLG [25], ODIN [24], BUTD-DETR [23]) appear as baselines and as implementation precedents, e.g., 'we apply voxel-based token merging (with a voxel size of 5cm) to the feature-coordinate pairs, following [24, 25, 58]' and 'we lift 2D data to 3D with probability p using reconstruction models such as MoGE [47] following UniVLG [25]'; these are empirical design choices, not load-bearing uniqueness theorems and not fitted inputs. The limitation passages in Section 5 ('we assume that this coordinate system is largely static over time'; 'Qwen-3D relies on externally estimated depth maps and camera poses') are acknowledged assumptions that bound generalizability but do not make any result equal to its input. No equation in the paper defines X in terms of Y and then claims to predict X. Consequently, no circular step is present; score 2 reflects only the presence of minor, non-load-bearing self-citations in baselines and implementation details.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The model is an empirical system, so the ledger lists the main hand-chosen hyperparameters and domain assumptions. No new physical or conceptual entities are introduced; the proposed 3D RoPE and dense token decoding are architectural methods, not entities.

free parameters (10)
  • voxel_size = 5cm
    Chosen for token merging in Section 3.1; affects the number of visual tokens and granularity of the scene representation.
  • number_of_object_queries N = not reported
    Number of learnable queries in the mask decoder (Section 3.3); a standard hyperparameter that affects capacity.
  • loss_weights alpha_mask, alpha_textground, alpha_gen = not reported
    Weights in Equation (1); their values are not given in the text, but they control the balance between segmentation, grounding, and text generation objectives.
  • LoRA rank = not reported
    Rank of the low-rank adaptation used to fine-tune the vision-language attention layers; not specified in Implementation Details.
  • learning_rate = 1e-4
    Reported in Implementation Details.
  • train_iterations = 200k
    Reported in Implementation Details.
  • effective_batch_size = 8
    Reported in Implementation Details.
  • frames_per_scene_training = 15
    Subsampling of training frames following prior work; affects redundancy and computation.
  • 3D RoPE frequency ranges = not specified numerically
    The paper says the same range of frequencies is used for each dimension but does not give the range; this is a key design choice for 3D positional encoding.
  • Hungarian cost weights = standard Mask2Former coefficients
    Chosen after finding alternative weighting from UniVLG hinders performance (Appendix 8.8).
assumptions (5)
  • domain assumption External depth maps and camera poses are available for all input frames.
    The entire 3D unprojection in Section 3.1 assumes supplied depth and pose, acknowledged in Section 5 Limitations.
  • domain assumption The 3D scene is static during the multi-view observation sequence.
    Explicitly identified in Section 5 as a limitation of the single persistent coordinate system.
  • ad hoc to paper Qwen2.5-VL's pretrained 2D visual features remain useful when repositioned by 3D coordinates.
    The paper freezes the ViT and only LoRA-tunes the VL attention, relying on transfer from 2D pretraining; this is not proven outside the reported benchmarks.
  • domain assumption 5cm voxel pooling preserves enough geometric detail for grounding and segmentation.
    Used in Section 3.1; the paper only varies frame count, not voxel size, so the sensitivity to this choice is unmeasured.
  • domain assumption A Mask2Former-style decoder can jointly reason over visual and language tokens to produce precise masks.
    The decoder is used as the grounding head; the paper relies on the architecture working without formal proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding." pith.science (2026). https://pith.science/paper/HUVIGOD3

@misc{pith2026260802980,
  author       = {Pith},
  title        = {Pith review of: Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUVIGOD3}},
  note         = {Machine review of arXiv:2608.02980}
}
read the original abstract

Large Multimodal Models (LMMs) have achieved remarkable success on images and short videos, yet scaling them to long videos remains challenging due to frame-centric tokenization and limited context windows. 3D geometry provides a natural compression mechanism for visual streams: depth and camera pose enable observations from multiple views and time steps to be fused into a persistent, world-aligned representation. While recent 3D LMMs leverage geometry-aware representations to improve spatial reasoning, they continue to lag behind specialist 3D perception systems on grounding and segmentation tasks. We argue that a key limitation is geometry-aware decoding: existing methods communicate 3D predictions through language tokens, proposal selection, or lightweight grounding queries, creating a bottleneck between language reasoning and dense geometric prediction. Building on these insights, we introduce Qwen-3D, a geometry-aware LMM that compresses visual information within the Qwen backbone using multi-view geometric cues, enabling efficient long-horizon visual reasoning over static scenes. Qwen-3D augments visual tokens with 3D Rotary Positional Embeddings, allowing attention to operate directly in 3D scene space rather than across independent image frames and thereby facilitating scalable cross-view and temporal reasoning. To bridge language and geometry, Qwen-3D incorporates a query-based segmentation decoder that grounds language directly in the underlying 3D scene representation, unifying referential grounding, instance segmentation, and visual question answering across both images and videos. Across a diverse set of benchmarks, Qwen-3D surpasses existing 3D LMMs and outperforms several large proprietary 2D models. Notably, Qwen-3D achieves these improvements while maintaining strong performance on standard 2D vision-language benchmarks by jointly training on 2D and 3D data.

Figures

Figures reproduced from arXiv: 2608.02980 by the authors.

Figure 1
Figure 1. Qwen-3D performs attention directly in 3D world space rather than over independent image frames. Given multi-view RGB observations, depth, and camera poses, Qwen-3D maps visual tokens into a shared 3D coordinate system and applies geometry-aware attention through 3D Rotary Positional Embeddings. The model jointly supports language reasoning, 2D grounding, and 3D grounding within a unified architecture, achieving sta… view at source ↗
Figure 2
Figure 2. Decoding object information in 3D LMMs. From left to right: (i) Text-space decoding, as in 3D-LLM [17], where object coordinates or bounding boxes are generated autoregressively as language tokens. Unlike images, 3D scenes do not admit a canonical world coordinate frame, making dense spatial prediction through text generation inherently ambiguous. (ii) Proposal-selection decoding, as in Video-3D-LLM [19, 20, 57], wh… view at source ↗
Figure 3
Figure 3. Qwen-3D architecture. Given a natural language query and multi-view RGB-D inputs, the Qwen2.5-VL vision encoder extracts multi-view 2D features, unprojects them into world-space XYZ coordinates, and voxel-pools them to reduce spatial redundancy. The resulting tokens are fused with text via Qwen vision–language attention layers augmented with 3D Rotary Positional Embeddings and LoRA adaptation. Two task heads operate… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Sensitivity analysis to camera pose noise (left) and depth noise (right) on ScanRefer (Top-1@0.25). We compare Qwen-3D [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity to number of views on ScanRefer (Top-1@0.25) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Failure cases of Qwen-3D on 3D grounding tasks. The red segmentation masks and boxes refer to Qwen-3D’s prediction and the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Visualizations of Qwen-3D’s predictions on [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Visualizations of Qwen-3D’s predictions on [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Visualizations of Qwen-3D’s instance segmentation predictions on [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Visualizations of Qwen-3D’s responses to visual question-answering tasks on [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 48 canonical work pages

  1. [1]

    ReferIt3D: Neural Listen- ers for Fine-Grained 3D Object Identification in Real-World Scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. ReferIt3D: Neural Listen- ers for Fine-Grained 3D Object Identification in Real-World Scenes. InProc. ECCV, 2020. 6, 13, 14, 15

    work page 2020

  2. [2]

    Llama 3: The llama-3 herd of models.https: //ai.meta.com/llama/, 2024

    Meta AI. Llama 3: The llama-3 herd of models.https: //ai.meta.com/llama/, 2024. Large language model. 7, 15

    work page 2024

  3. [3]

    Locate 3d: Real-world ob- ject localization via self-supervised learning in 3d, 2025

    Sergio Arnaud, Paul McVay, Ada Martin, Arjun Majum- dar, Krishna Murthy Jatavallabhula, Phillip Thomas, Rus- lan Partsey, Daniel Dugas, Abha Gejji, Alexander Sax, Vincent-Pierre Berges, Mikael Henaff, Ayush Jain, Ang Cao, Ishita Prasad, Mrinal Kalakrishnan, Michael Rabbat, Nicolas Ballas, Mido Assran, Oleksandr Maksymets, Aravind Ra- jeswaran, and Franzis...

    work page 2025

  4. [4]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129– 19139, 2022. 6, 7, 8, 13

    work page 2022

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, and Jialin Wang et. al. Qwen2.5-vl technical report, 2025. 2, 3, 4, 8, 14

    work page 2025

  6. [6]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. InInternational Conference on Learning Representations, 2023. 16

    work page 2023

  7. [7]

    From thousands to billions: 3d visual language grounding via render-supervised distillation from 2d vlms, 2025

    Ang Cao, Sergio Arnaud, Oleksandr Maksymets, Jian- ing Yang, Ayush Jain, Sriram Yenamandra, Ada Martin, Vincent-Pierre Berges, Paul McVay, Ruslan Partsey, Aravind Rajeswaran, Franziska Meier, Justin Johnson, Jeong Joon Park, and Alexander Sax. From thousands to billions: 3d visual language grounding via render-supervised distillation from 2d vlms, 2025. 6

    work page 2025

  8. [8]

    End- to-End Object Detection with Transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-End Object Detection with Transformers. InProc. ECCV,

    work page

Show all 62 references
  1. [9]

    Matterport3d: Learning from rgb-d data in indoor environments.arXiv preprint arXiv:1709.06158, 2017

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments.arXiv preprint arXiv:1709.06158, 2017. 6

    2017 arXiv

  2. [10]

    ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language

    Dave Zhenyu Chen, Angel Chang, and Matthias Nießner. ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language. InProc. ECCV, 2020. 6, 7, 13, 15

    2020

  3. [11]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InProceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2024

  4. [12]

    Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Run- sen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024. 1, 3, 4, 5, 6, 7, 8, 15

    2024 arXiv

  5. [13]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022. 5, 6, 16

    2022

  6. [14]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 1, 6, 8

    2017

  7. [15]

    Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration.ACM Transactions on Graphics 2017 (TOG),

    Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration.ACM Transactions on Graphics 2017 (TOG),

    2017

  8. [16]

    Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024. 3, 7, 15

    2024 arXiv

  9. [17]

    3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- jecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 6

    2021

  11. [19]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37: 113991–1...

    2024

  12. [20]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 1, 3, 4, 6, 7, 15

    2023 arXiv

  13. [21]

    Revisiting multimodal positional encoding in vision-language models, 2026

    Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang, Junyang Lin, and Shuai Bai. Revisiting multimodal positional encoding in vision-language models, 2026. 16

    2026

  14. [22]

    Reason3d: Searching and reasoning 3d segmentation via large language model.3DV, 2025

    Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model.3DV, 2025. 4, 6, 7, 15

    2025

  15. [23]

    Bottom up top down detection transform- ers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Kate- rina Fragkiadaki. Bottom up top down detection transform- ers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417–433. Springer, 2022. 4, 6, 7, 15

    2022

  16. [24]

    Odin: A single model for 2d and 3d segmentation

    Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d segmentation. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, p...

    2024

  17. [25]

    Unifying 2d and 3d vision-language un- derstanding, 2025

    Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Ar- naud, Ada Martin, Alexander Sax, Franziska Meier, and Ka- terina Fragkiadaki. Unifying 2d and 3d vision-language un- derstanding, 2025. 1, 4, 6, 7, 8, 13, 14, 15, 16

    2025

  18. [26]

    MDETR - Modulated Detection for End-to-End Multi-Modal Under- standing

    Aishwarya Kamath, Mannat Singh, Yann André LeCun, Is- han Misra, Gabriel Synnaeve, and Nicolas Carion. MDETR - Modulated Detection for End-to-End Multi-Modal Under- standing. InProc. ICCV, 2021. 6

    2021

  19. [27]

    ReferItGame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in pho- tographs of natural scenes. InProceedings of the 2014 Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 787–798, Doha, Qatar, 2014. Associ...

    2014

  20. [28]

    Restr: Convolution-free referring image segmentation using transformers, 2022

    Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution-free referring image segmentation using transformers, 2022. 8

    2022

  21. [29]

    Mask-attention-free transformer for 3d in- stance segmentation

    Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention-free transformer for 3d in- stance segmentation. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 3693–3703,

  22. [30]

    Lisa: Reasoning segmenta- tion via large language model, 2024

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model, 2024. 4

    2024

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 3

    2023

  24. [32]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2022

  25. [33]

    3eed: Ground everything everywhere in 3d

    Rong Li, Yuhao Dong, Tianshuai Hu, Ao Liang, Youquan Liu, Dongyue Lu, Liang Pan, Lingdong Kong, Junwei Liang, and Ziwei Liu. 3eed: Ground everything everywhere in 3d. InNeurIPS 2025 Datasets and Benchmarks Track, 2025. 13

    2025

  26. [34]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  27. [35]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 6, 8, 13

    2023

  28. [36]

    View-on- graph: Zero-shot 3d visual grounding via vision-language reasoning on scene graphs

    Yuanyuan Liu, Haiyang Mei, Li Zhang, et al. View-on- graph: Zero-shot 3d visual grounding via vision-language reasoning on scene graphs. InProceedings of the AAAI Con- ference on Artificial Intelligence (AAAI), 2026. 7

    2026

  29. [37]

    3d-sps: Single-stage 3d visual grounding via referred point progressive selection

    Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR). IEEE, 2022. 4

    2022

  30. [38]

    Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022. 6, 8, 13

    2022 arXiv

  31. [39]

    Goucher, Adam Perelman, Aditya Ramesh, and Aidan Clark et al

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, and Aidan Clark et al. Gpt-4o system card, 2024. 3, 7, 15

    2024

  32. [40]

    Languagerefer: Spatial-language model for 3d visual grounding

    Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial-language model for 3d visual grounding. InConference on Robot Learning, pages 1046–

  33. [41]

    Language- grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language- grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision, pages 125–141. Springer, 2022. 1, 6, 8, 13

    2022

  34. [42]

    Mask3d: Mask trans- former for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask trans- former for 3d semantic instance segmentation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023. 8, 16

    2023

  35. [43]

    Evaluating zero-shot gpt-4v performance on 3d vi- sual question answering benchmarks, 2024

    Simranjit Singh, Georgios Pavlakos, and Dimitrios Sta- moulis. Evaluating zero-shot gpt-4v performance on 3d vi- sual question answering benchmarks, 2024. 7

    2024

  36. [44]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.https://crfm.stanford.edu/2023/03/ 13/alpaca.html, 2023. Dataset and model release. 6, 13

    2023

  37. [45]

    Gemini: A family of highly capable multimodal models, 2025

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, and Radu Soricut et al. Gemini: A family of highly capable multimodal models, 2025. 3

    2025

  38. [46]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, and Jinze Bai et. al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024. 7, 15

    2024

  39. [47]

    Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision.arXiv preprint arXiv:2410.19115, 2024

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision.arXiv preprint arXiv:2410.19115, 2024. 6

    arXiv

  40. [48]

    Realworldqa

    X.AI. Realworldqa. Blog post, 2024. Accessed: 2025-05-12. 8

    2024

  41. [49]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, and Bo Zheng et. al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024. 7, 15

    2024 arXiv

  42. [50]

    Sat: 2d semantics assisted training for 3d visual grounding

    Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856–1866, 2021. 15

    2021

  43. [51]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip H. S. Torr. Lavt: Language-aware vision transformer for referring image segmentation, 2022. 8

    2022

  44. [52]

    Scannet++: A high-fidelity dataset of 3d in- door scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d in- door scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023. 7

    2023

  45. [53]

    Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual refer- ring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual refer- ring. InProceedings of the IEEE/CVF International Conf...

    2021

  46. [54]

    Multi3drefer: Grounding text description to multiple 3d ob- jects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d ob- jects. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15225–15236, 2023. 14

    2023

  47. [55]

    Towards learning a generalist model for embod- ied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Li- wei Wang. Towards learning a generalist model for embod- ied navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13624– 13634, 2024. 7, 15

    2024

  48. [56]

    Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors.arXiv preprint arXiv:2505.24625,

    Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors.arXiv preprint arXiv:2505.24625,

  49. [57]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8995–9006, 2025. 1, 3, 4, 5, 6, 7, 13, 14, 15

    2025

  50. [58]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness, 2024. 1, 3, 4, 5, 6, 7, 8, 13, 14, 15

    2024

  51. [59]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 1, 4, 6, 7, 15

    2023

  52. [60]

    Unifying 3d vision-language understanding via prompt- able queries.arXiv preprint arXiv:2405.11442, 2024

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via prompt- able queries.arXiv preprint arXiv:2405.11442, 2024. 4, 6, 7, 8, 15

    2024 arXiv

  53. [61]

    Generalized decoding for pixel, image, and lan- guage

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and lan- guage. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 1...

  54. [62]

    Appendix 8.1. Sensitivity to Pose and Depth Noise While Qwen-3D already operates on real-world sensor noise in all experiments reported in the paper, we further stress-test the model under controlled settings with substan- tial depth and camera pose noise. Following UniVLG [25...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.