Pith. sign in

REVIEW 6 major objections 7 minor 1 cited by

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

T0 review · 6 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Token-level sparse routing over five 3D modalities delivers state-of-the-art scene understanding on ScanNet benchmarks.

desk verdict Solid systems paper on fusing five 3D modalities with sparse MoE routing; the benchmark gains are real and the ablation logic is clean, but the headline claim of question-type-aware routing is under-supported and the results rest on a single data family. read the letter →

arxiv 2505.21079 v1 pith:HS3RHO2L submitted 2025-05-27 cs.CV

Yue Zhang , Yingzhao Jian , Hehe Fan , Yi Yang , Roger Zimmermann This is my paper
classification cs.CV
keywords 3Dsceneunderstandingmixtureofexpertsmultimodallargelanguagemodeladaptivemodalityfusiontoken-levelroutingquestionansweringvisualgroundingdensecaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a 3D vision-language model understands indoor scenes better when it ingests all five major 3D representations at once—multi-view RGB and depth images, bird's-eye-view maps, point clouds, and voxel grids—and lets a learned router send each token to a specialist expert inside the language model. The motivation is that different questions need different evidence: color questions should lean on RGB tokens, shape questions on point clouds, counting and location on geometric modalities. The authors report that this design outperforms prior models on ScanQA, SQA3D, ScanRefer, Multi3DRefer, and Scan2Cap, all built from the same ScanNet scans, and that routing patterns visibly shift with question type. If the claim holds, a single architecture can replace the usual fixed choice of one or two input modalities, and capacity can scale by adding experts rather than densifying the whole network.

What carries the argument

The central object is the token-level sparse Mixture-of-Experts (MoE) layer: a small set of replicated feed-forward experts inside the language model, each paired with a learnable router that scores token $f_i$ against expert $E_e$ as $s_i^{(e)} = w_e^\top f_i$, normalizes the scores into probabilities $\pi_i^{(e)}$, and combines only the top-$k$ experts' outputs. The routing is what converts a fixed concatenation of five modalities into an adaptive fusion, because the same modality token can be processed by different experts depending on the accompanying question. A sparsity-aware balancing loss keeps the eight experts per layer from collapsing onto a few popular routes, and the paper shows that the learned routes differ systematically by modality and by question category.

What would settle it

A decisive experiment would be to take the trained model, freeze all expert weights, and reroute color-question RGB tokens to the experts normally preferred by point-cloud tokens: if the model's accuracy on color questions does not fall, then the claimed query-aware routing is not load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that sparse soft routing at the token level produces adaptive multimodal fusion: each token $f_i$ is scored against $E=8$ experts by $s_i^{(e)} = w_e^\top f_i$, the scores are normalized into routing probabilities $\pi_i^{(e)}$, and the top-$k$ (with $k=2$) outputs are combined as $\hat{f}_i = \sum_{e \in S_i} \pi_i^{(e)} E_e(f_i)$. Installed in six layers of a 7-billion-parameter language model, this design yields the numbers the paper reports as state of the art on its benchmarks: 30.8 EM@1 on ScanQA, 57.2 EM@1 on SQA3D, 62.7/57.4 Acc@0.25/0.5 on ScanRefer, 65.1/60.5 F1@0.25/0.5 on Multi3DRefer, and 85.2 CIDEr@0.5 on Scan2Cap. The distinctive observational claim is that the router becomes query-aware: color questions route RGB tokens mainly to one expert, shape questions route point-cloud tokens to another, and these preferences differ by question category in the paper's visualizations. The paper further claims the MoE module itself drives part of the gain, since replacing the dense feed-forward network with experts raises ScanQA EM@1 from 27.3 to 30.8 and CIDEr from 88.4 to 97.6.

Load-bearing premise

The load-bearing premise is that the reported gains come from the learned routing rather than from the added expert parameters or from statistical quirks of the single ScanNet-derived benchmark family, since the paper does not evaluate other scene types or disable routing to prove causation.

Editorial extensions

If this is right

  • A single model can serve question answering, dense captioning, and visual grounding from the same five-modality input, so task-specific input pipelines are not needed.
  • Capacity can be scaled by adding experts: the paper's ablation on ScanQA moves EM@1 from 27.3 without MoE to 29.5 with 4 experts, 29.9 with 6, and 30.8 with 8.
  • The router provides an inspectable map of which modality the model treats as evidence for a question type, which can guide sensor selection for new 3D tasks.
  • Modality ablations assign clear roles—RGB for color, point clouds for counting and shape, voxels and BEV for location—so future systems can prune or emphasize modalities according to the question distribution they face.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation covers only benchmarks derived from one indoor scan family, so an unstated test of the adaptive-fusion claim is whether the same recipe transfers to outdoor or object-centric 3D scenes with different modality statistics.
  • Because the router is trained only with a load-balancing loss and no direct supervision of modality-expert affinity, the observed query-aware patterns suggest routing entropy could be used at inference as an uncertainty signal for answers; the paper does not explore that.
  • The authors' own limitations section notes that token budgets force viewpoint pruning and point-cloud downsampling, so the claimed gains are demonstrated under constraints that can omit fine details or critical viewpoints.
  • The two-stage recipe—align modalities with a dense LLM first, then swap in MoE—looks reusable for adding a sixth modality later, since the new tokens only need an adapter before the router learns their affinities; the paper demonstrates five, not the ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper proposes Uni3D-MoE, a 3D multimodal large language model that fuses five ScanNet-style modalities (multi-view RGB, RGB-D, BEV, point cloud, and voxel) through modality-specific adapters and sparse mixture-of-experts layers inserted into a LLaVA-v1.5-7B backbone. The training is two-stage: first a dense alignment stage with LoRA, then an MoE stage in which the router and expert modules are trained while other parameters are frozen. The central claims are (i) state-of-the-art results on ScanQA, SQA3D, ScanRefer, Multi3DRefer, and Scan2Cap; (ii) complementary roles for each modality; and (iii) a learnable soft router that performs query-aware modality-expert specialization, e.g., color questions route RGB tokens to expert E4 and shape questions route point-cloud tokens to expert E3. The experiments include main-table comparisons, per-question-type modality ablations, MoE ablations over expert counts and layer placements, and routing visualizations.

Significance. The empirical structure is largely sound: the single-modality, leave-one-out, and MoE ablations are well designed, and the benchmark numbers are broadly consistent across the main tables and appendix. If the query-aware routing claim were established, this would be a useful contribution to multimodal 3D scene understanding. The paper also provides positive evidence for complementary modality roles and for MoE gains on several curated question categories and on ScanQA. However, the Scan2Cap state-of-the-art claim is contradicted by the paper's own appendix, the query-aware routing evidence is confounded and does not follow from the routing equation, and the reported relative gains are numerically inconsistent with Table 1. The absence of error bars further weakens claims that rest on sub-point margins. These issues are fixable with additional analysis and corrected reporting, so the appropriate outcome is a major revision rather than rejection.

major comments (6)
  1. [Sec. 4.2, Table 1 vs Appendix Table 9] The main text states that on Scan2Cap Uni3D-MoE 'achieves an improvement of 5.8% on CIDEr@0.5 compared to the previous advanced method PQ3D,' and Table 1 lists 85.2 as the best CIDEr@0.5. Appendix Table 9, however, reports GPT4Scene-HDM at 86.3 CIDEr@0.5, which is higher than the reported 85.2. The paper therefore does not currently hold the Scan2Cap state-of-the-art that the abstract and main text imply. Please clarify the evaluation protocol (e.g., different IoU or filtering) or revise the headline claim; if GPT4Scene-HDM was evaluated under different settings, that must be stated in the main text.
  2. [Sec. 3.2, Eq. (1); Sec. 4.3, Fig. 5] The central claim of query-aware routing is not established. In Eq. (1), the routing probability for token f_i is computed as softmax over w_e^T f_i, so a visual token's expert choice is a function of that token alone at the routing step. A question could influence visual token features only indirectly through preceding self-attention, but the paper provides no analysis of that mechanism. Fig. 5 normalizes routing frequencies within each modality, which controls for token-count imbalance but not for scene distributions, object-class co-occurrence, or captioning style across question categories. A concrete test would be to hold the visual input fixed and vary only the question, then compare routing distributions; alternatively, condition the router explicitly on pooled question features and ablate that conditioning. As written, the 'question-type aware' conclusion is a confounded interpretation of the routing visualizations.
  3. [Sec. 4.2, relative-gain claims] The reported relative improvements over state-of-the-art do not match Table 1. Taking the previous best values in Table 1 (LLaVA-3D: EM@1 27.0, BLEU-4 14.5, CIDEr 91.7), the relative gains for Uni3D-MoE are approximately 14.1% on EM@1, 20.7% on BLEU-4, and 6.4% on CIDEr, not the stated 11.7%, 16.8%, and 6.0%. Please state explicitly which baseline each relative gain is computed against and recompute all values; as written, the numbers are internally inconsistent.
  4. [Tables 1, 5, 6, 8, 9] No error bars, multiple seeds, or significance tests are reported anywhere, and several headline margins are sub-point. For example, ScanRefer Acc@0.25 is 62.7 vs. GPT4Scene-HDM's 62.6, and Multi3DRefer F1@0.25 is 65.1 vs. 64.5. Given the small margins in the grounding tables, seed variance could change the ranking. Please report results over at least three seeds with standard deviations, or provide a significance analysis for the claimed SOTA comparisons.
  5. [Sec. 4.4 vs Appendix Tables 11-13] The MoE ablation across benchmarks shows that MoE integration decreases CIDEr on Scan2Cap at both IoU thresholds (91.7 to 89.9 at IoU@0.25 and 86.3 to 85.2 at IoU@0.5), and gives only 0.1/0.4 gains on ScanRefer. The main text says 'incorporating the MoE module consistently improves performance across all these categories,' but that statement refers only to the curated question-type set in Table 3. The mixed results on the official Scan2Cap metric should be presented in the main text and reconciled with the abstract's framing of MoE as a source of consistent improvement.
  6. [Sec. 4.1, Sec. 4.5, Appendix D] All experiments are conducted on ScanNet-derived corpora, as described in Sec. 4.1 and the appendix. The paper frames the method as a scalable and comprehensive 3D scene understanding system, but no evidence is provided outside this single dataset family with correlated annotation statistics. The limitations sections acknowledge token budget and annotation quality but do not mention this scope restriction. Please either add cross-dataset evidence (e.g., Matterport3D, nuScenes, or another 3D dataset family) or explicitly narrow the claims to ScanNet-style indoor scenes.
minor comments (7)
  1. [Throughout] There are several typographical issues, most notably the spaced 'V oxel' in Sec. 3.1, Fig. 2, and Algorithm 1.
  2. [Sec. 4.2] The text cites 'the ScanQA benchmark [19]' but ScanQA is reference [18]; the citation should be corrected.
  3. [Table 8] The table formatting is broken: the 'Task-Specific Model' header and the first several rows lack proper line separation, and some entries are not aligned with their columns.
  4. [Sec. 4.3, Fig. 6] The phrase 'previously unseen RGB and BEV tokens' is unclear; please specify what 'unseen' means in this context (e.g., held-out scenes or non-training samples).
  5. [Fig. 5] The line-thickness visualization is qualitative; please report numerical routing proportions or a table to support the claimed preferences for specific modality-expert pairs.
  6. [Contributions, Sec. 1] The 'first' claims should be softened given that 3D-MoE [52] and MiniGPT-3D [53] already apply MoE to 3D tasks; the novelty should be positioned as the unified multi-modality design rather than the first use of MoE in 3D.
  7. [Algorithm 1] The variable Vscene is defined as voxels whose type is in {floor, ceiling, wall}, but the text says the algorithm removes low-contribution segments such as the floor; please clarify whether these voxels are kept or pruned.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark gains and ablations are measured on held-out data; the router analysis is interpretive, not a fitted prediction.

full rationale

I walked the claimed derivation chain. The central predictive claims are the held-out benchmark numbers in Tables 1, 5, 6, 7, 8, and 9 (ScanQA, SQA3D, ScanRefer, Multi3DRefer, Scan2Cap), all measured with standard metrics on evaluation splits rather than derived from fitted parameters. The MoE contribution is quantified by comparing a dense Stage-I baseline (w/o MoE) against the Stage-II MoE model under identical training data (Tables 3, 4, 10-13); this is an empirical ablation, not a renamed fit. The modality contribution analysis (Table 2) is leave-one-out measurement on question-type subsets, and no equation in Sec. 3.2 defines those gains in terms of the inputs. The routing mechanism (Eq. 1) is a learned per-token softmax over hidden states; Fig. 5 is an interpretation of routing frequencies, not a quantity forced by construction. The paper contains no load-bearing self-citation chain: its cited MoE and 3D MLLM baselines are external prior work, and no uniqueness theorem or ansatz is imported from the authors' own earlier papers. The limitation statement in Sec. 4.5 concerns token budgets and dataset quality, which is an honest scope caveat rather than evidence of circularity. The question of whether Fig. 5 fully controls for token-count imbalance or demonstrates true query-aware routing is an evidence-quality and interpretation concern, not a circular-derivation concern.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are introduced beyond the standard MoE machinery: each expert is a replicated FFN, the router is a learned linear scoring function, and all encoders are borrowed from prior work. The only novel algorithmic component is the MVCS keyframe sampler, which is a heuristic preprocessing method, not an invented entity. The four free parameters listed are standard hyperparameters; none is fit to the central benchmark claim.

free parameters (4)
  • MoE load-balancing coefficient alpha = 0.01
    Set following MoE-LLaVA and Uni-MoE, per Sec. 4.1. Not fit to the evaluation benchmarks; a standard hyperparameter choice.
  • Number of selected views V in MVCS = 24
    Chosen design budget for token efficiency, per Sec. F.1. Likely tuned by validation; effect on accuracy not ablated standalone.
  • Number of MoE layers and expert count = Layers [8,12,16,20,24,28], 8 experts, top-2
    The layer choice is explicitly ablated in Table 10; the paper reports the best configuration. Expert count 8 is ablated in Table 4 as the best of {4,6,8}. These are architecture hyperparameters selected by validation, not fit to the test claim.
  • LoRA rank and alpha = r=32, alpha=64, dropout=0.05
    Rank 32 with alpha 64 is a common LoRA recipe (Sec. F.2); no evidence of dataset-specific tuning.
assumptions (4)
  • domain assumption The ScanNet-derived training mixture (dense captioning, QA, visual grounding) is a sufficient instruction-tuning corpus to learn 3D scene understanding that transfers to the held-out splits of the same benchmarks.
    Invoked throughout the experimental setup (Sec. 4.1) and is the single-family benchmark scope underlying every reported result.
  • domain assumption Instance segmentation masks (e.g., from Mask3D) provide consistent object IDs across all modality views, prompts, and tasks.
    Used for voxel segment-wise pooling, BEV semantic cues, ScanRefer/Multi3DRefer object IDs, and Scan2Cap captions (Secs. 3.1 and F.1); errors here propagate through all grounding and captioning metrics.
  • domain assumption Pretrained encoders (DINOv2, CLIP, PointNet++, Mask3D) provide features that are already aligned enough for the lightweight adapters to map them into the LLM embedding space.
    Assumed in Sec. 3.1 and F.1; the paper does not ablate encoder choices or verify alignment quality independently of downstream task accuracy.
  • domain assumption Expert specialization can be induced by replicating the LLM's FFN with a learned router, without requiring expert-specific training data.
    Stage II initialization and trainable components (router plus experts) follow MoE-LLaVA (Secs. 3.3 and F.2); the paper relies on the assumption that optimization alone produces useful specialization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts." pith.science (2026). https://pith.science/paper/HS3RHO2L

@misc{pith2026250521079,
  author       = {Pith},
  title        = {Pith review of: Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HS3RHO2L}},
  note         = {Machine review of arXiv:2505.21079}
}
read the original abstract

Recent advancements in multimodal large language models (MLLMs) have demonstrated considerable potential for comprehensive 3D scene understanding. However, existing approaches typically utilize only one or a limited subset of 3D modalities, resulting in incomplete representations of 3D scenes and reduced interpretive accuracy. Furthermore, different types of queries inherently depend on distinct modalities, indicating that uniform processing of all modality tokens may fail to effectively capture query-specific context. To address these challenges, we propose Uni3D-MoE, a sparse Mixture-of-Experts (MoE)-based 3D MLLM designed to enable adaptive 3D multimodal fusion. Specifically, Uni3D-MoE integrates a comprehensive set of 3D modalities, including multi-view RGB and depth images, bird's-eye-view (BEV) maps, point clouds, and voxel representations. At its core, our framework employs a learnable routing mechanism within the sparse MoE-based large language model, dynamically selecting appropriate experts at the token level. Each expert specializes in processing multimodal tokens based on learned modality preferences, thus facilitating flexible collaboration tailored to diverse task-specific requirements. Extensive evaluations on standard 3D scene understanding benchmarks and specialized datasets demonstrate the efficacy of Uni3D-MoE.

Figures

Figures reproduced from arXiv: 2505.21079 by the authors.

Figure 1
Figure 1. Challenges in 3D scene understanding. (1) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our method. Uni3D-MoE covers major input modalities of 3D scenes, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of two-stage training strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Token-to-expert routing across MoE layers. The first two rows show the modality token [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Modality-expert routing preferences across different question types. Line thick￾ness indicates normalized token routing pro￾portions. Preferred modality-expert routes for each query type are highlighted in color; oth￾ers are shown in gray. Question-type Aware Expert Sp…
Figure 6
Figure 6. Figure 6: Top-1 activated routing pathways for differ [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Visualization of Uni3D-MoE performing diverse 3D scene understanding tasks. Examples [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: illustrates the failure cases of our method. In the first example, given the query “a circular end table, it is next to a teal couch” the model predicts object 13, while the ground truth is object 21. Notably, object 13 also accurately satisfies the description, as it …
Figure 10
Figure 10. Figure 10: Top-10 activated routing pathways for different modalities, highlighting dynamic and [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Expert assignment distribution for each modality across MoE layers in Uni3D-MoE, [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Multimodal dialogue data format [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Prompt template. Multi-view Depth Encoder. Inspired by [33], we leverage depth images D ∈ R V ×H×W×1 and camera parameters to back-project each 2D patch into 3D space, obtaining its corresponding 3D position. The 2D patch tokens are first extracted from multi-view RGB…
Figure 14
Figure 14. Figure 14: Prompt Template for different tasks. B ∈ R H×W×3 , where H and W are the height and width, respectively. To enhance object-level understanding, we incorporate instance segmentation into the BEV using numeric labels and colored regions, offering explicit semantic cues.…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A multimodal language model that dynamically routes queries to the most relevant scene modalities and modality-specialized experts, achieving state-of-the-art results on five 3D benchmarks.

Reference graph

Works this paper leans on

93 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hierar- chical open-vocabulary 3d scene graphs for language-grounded robot navigation

    Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hierar- chical open-vocabulary 3d scene graphs for language-grounded robot navigation. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024

    2024

  2. [2]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. Advances in Neural Information Processing Systems, 37:5285–5307, 2024

    2024

  3. [3]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024

    2024

  4. [4]

    Multi-modal data-efficient 3d scene understanding for autonomous driving

    Lingdong Kong, Xiang Xu, Jiawei Ren, Wenwei Zhang, Liang Pan, Kai Chen, Wei Tsang Ooi, and Ziwei Liu. Multi-modal data-efficient 3d scene understanding for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  5. [5]

    Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving

    Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jianbing Shen. Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3347–3355, 2025

    2025

  6. [6]

    Drivinggaus- sian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes

    Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaus- sian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21634–21643, 2024

    2024

  7. [7]

    Editable scene simulation for autonomous driving via collaborative llm-agents

    Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative llm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15077–15087, 2024

    2024

  8. [8]

    How to enable llm with 3d capacity? a survey of spatial reasoning in llm

    Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable llm with 3d capacity? a survey of spatial reasoning in llm. arXiv preprint arXiv:2504.05786, 2025

    arXiv 2025

Show all 93 references
  1. [9]

    Scenecraft: An llm agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. In Forty-first International Conference on Machine Learning, 2024

    2024

  2. [10]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  3. [11]

    Grounded 3d-llm with referent tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370, 2024

    2024 arXiv

  4. [12]

    Comp4d: Llm-guided compositional 4d scene generation

    Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: Llm-guided compositional 4d scene generation. arXiv preprint arXiv:2403.16993, 2024

    2024 arXiv

  5. [13]

    Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 76...

    2024

  6. [14]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023

    2023 arXiv

  7. [15]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Pr...

    2024

  8. [16]

    Gpt4scene: Understand 3d scenes from videos with vision-language models

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025

    2025 arXiv

  9. [17]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. arXiv preprint arXiv:2412.00493, 2024

    2024 arXiv

  10. [18]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022. 10

    2022

  11. [19]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203, 2021

    2021

  12. [20]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022

    2022 arXiv

  13. [21]

    Pointllm: Empower- ing large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In European Conference on Computer Vision, pages 131–147. Springer, 2024

    2024

  14. [22]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arX...

    2023 arXiv

  15. [23]

    Uni3d-llm: Unifying point cloud perception, generation and editing with large language models

    Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arXiv preprint arXiv:2402.03327, 2024

    2024 arXiv

  16. [24]

    Objvariantensemble: Advancing point cloud llm evaluation in chal- lenging scenes with subtly distinguished objects

    Qihang Cao and Huangxun Chen. Objvariantensemble: Advancing point cloud llm evaluation in chal- lenging scenes with subtly distinguished objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1944–1952, 2025

    1944

  17. [25]

    Gpt4point: A unified framework for point-language understanding and generation

    Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26417–...

    2024

  18. [26]

    Unifying 3d vision-language understanding via promptable queries

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. In European Conference on Computer Vision, pages 188–206. Springer, 2024

    2024

  19. [27]

    Lidar-llm: Exploring the potential of large language models for 3d lidar understanding

    Senqiao Yang, Jiaming Liu, Renrui Zhang, Mingjie Pan, Ziyu Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Hongsheng Li, Yandong Guo, et al. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. In Proceedings of the AAAI Conference on Artificial Intell...

    2025

  20. [28]

    Kestrel: Point grounding multimodal llm for part-aware 3d vision-language understanding

    Junjie Fei, Mahmoud Ahmed, Jian Ding, Eslam Mohamed Bakr, and Mohamed Elhoseiny. Kestrel: Point grounding multimodal llm for part-aware 3d vision-language understanding. arXiv preprint arXiv:2405.18937, 2024

    2024 arXiv

  21. [29]

    Liba: Language instructed multi-granularity bridge assistant for 3d visual grounding

    Yuan Wang, Ya-Li Li, WU Eastman ZY , and Shengjin Wang. Liba: Language instructed multi-granularity bridge assistant for 3d visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8114–8122, 2025

    2025

  22. [30]

    4d-bench: Benchmarking multi-modal large language models for 4d object understanding

    Wenxuan Zhu, Bing Li, Cheng Zheng, Jinjie Mai, Jun Chen, Letian Jiang, Abdullah Hamdi, Sara Rojas Martinez, Chia-Wen Lin, Mohamed Elhoseiny, et al. 4d-bench: Benchmarking multi-modal large language models for 4d object understanding. arXiv preprint arXiv:2503.17827, 2025

    2025 arXiv

  23. [31]

    Space3d- bench: Spatial 3d question answering benchmark

    Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, and Marc Pollefeys. Space3d- bench: Spatial 3d question answering benchmark. arXiv preprint arXiv:2408.16662, 2024

    2024 arXiv

  24. [32]

    Embodied intelligence for 3d understanding: A survey on 3d scene question answering

    Zechuan Li, Hongshan Yu, Yihao Ding, Yan Li, Yong He, and Naveed Akhtar. Embodied intelligence for 3d understanding: A survey on 3d scene question answering. arXiv preprint arXiv:2502.00342, 2025

    2025 arXiv

  25. [33]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024

    2024 arXiv

  26. [34]

    3ur-llm: An end-to-end multimodal large language model for 3d scene understanding

    Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, and Huchuan Lu. 3ur-llm: An end-to-end multimodal large language model for 3d scene understanding. arXiv preprint arXiv:2501.07819, 2025

    2025 arXiv

  27. [35]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289–310. Springer, 2024

  28. [36]

    Image as a foreign language: Beit pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on...

    2023

  29. [37]

    Uni3dl: A unified model for 3d vision- language understanding

    Xiang Li, Jian Ding, Zhaoyang Chen, and Mohamed Elhoseiny. Uni3dl: A unified model for 3d vision- language understanding. In European Conference on Computer Vision, pages 74–92. Springer, 2024

    2024

  30. [38]

    Vision-language pre-training with object contrastive learning for 3d scene understanding

    Taolin Zhang, Sunan He, Tao Dai, Zhi Wang, Bin Chen, and Shu-Tao Xia. Vision-language pre-training with object contrastive learning for 3d scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7296–7304, 2024

    2024

  31. [39]

    When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models

    Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:...

    2024

  32. [40]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022

    2022

  33. [41]

    A survey on mixture of experts

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024

    2024 arXiv

  34. [42]

    Revisiting moe and dense speed-accuracy comparisons for llm training

    Xianzhi Du, Tom Gunter, Xiang Kong, Mark Lee, Zirui Wang, Aonan Zhang, Nan Du, and Ruoming Pang. Revisiting moe and dense speed-accuracy comparisons for llm training. arXiv preprint arXiv:2405.15052, 2024

    2024 arXiv

  35. [43]

    Promoe: Fast moe-based llm serving using proactive caching

    Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. Promoe: Fast moe-based llm serving using proactive caching. arXiv preprint arXiv:2410.22134, 2024

    2024 arXiv

  36. [44]

    Openmoe: An early effort on open mixture-of-experts language models

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024

    2024 arXiv

  37. [45]

    Vlmo: Unified vision-language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. O...

    2022

  38. [46]

    Scaling vision- language models with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision- language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023

    2023 arXiv

  39. [47]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. CoRR, abs/2401.15947, 2024

    2024 arXiv

  40. [48]

    Ada-k routing: Boosting the efficiency of moe-based llms

    Tongtian Yue, Longteng Guo, Jie Cheng, Xuange Gao, Hua Huang, and Jing Liu. Ada-k routing: Boosting the efficiency of moe-based llms. In The Thirteenth International Conference on Learning Representations, 2024

    2024

  41. [49]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  42. [50]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15913–15923, 2024

    2024

  43. [51]

    Uni-moe: Scaling unified multimodal llms with mixture of experts

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal llms with mixture of experts. IEEE Trans. Pattern Anal. Mach. Intell., 47(5):3424–3439, 2025

    2025

  44. [52]

    3d-moe: A mixture-of-experts multi-modal llm for 3d vision and pose diffusion via rectified flow

    Yueen Ma, Yuzheng Zhuang, Jianye Hao, and Irwin King. 3d-moe: A mixture-of-experts multi-modal llm for 3d vision and pose diffusion via rectified flow. arXiv preprint arXiv:2501.16698, 2025

    2025 arXiv

  45. [53]

    Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6617–6626, 2024

    2024

  46. [54]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  47. [55]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  48. [56]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017

    2017

  49. [57]

    Mask3d: Mask transformer for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023

    2023

  50. [58]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    2017

  51. [59]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020

    2020

  52. [60]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225– 15236, 2023

    2023

  53. [61]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  54. [62]

    Ross3d: Reconstructive visual instruction tuning with 3d-awareness

    Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. arXiv preprint arXiv:2504.01901, 2025

    2025 arXiv

  55. [63]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  56. [64]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  57. [65]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004

    2004

  58. [66]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566–4575, 2015

    2015

  59. [67]

    Context-aware alignment and mutual masking for 3d-language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3d-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10984–10994, 2023

    2023

  60. [68]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023

    2023

  61. [69]

    Internvl- x: Advancing and accelerating internvl series with efficient visual token compression

    Dongchen Lu, Yuyao Sun, Zilu Zhang, Leping Huang, Jianliang Zeng, Mao Shu, and Huo Cao. Internvl- x: Advancing and accelerating internvl series with efficient visual token compression. arXiv preprint arXiv:2503.21307, 2025

    2025 arXiv

  62. [70]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  63. [71]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  64. [72]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36:26...

    2023

  65. [73]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    2023

  66. [74]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. CoRR, abs/2312.08168, 2023. 13

    2023 arXiv

  67. [75]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  68. [76]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2024

    2024 arXiv

  69. [77]

    Principal components analysis (pca)

    Andrzej Ma ´ckiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19(3):303–342, 1993

    1993

  70. [78]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16464–16473, 2022

    2022

  71. [79]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11124–11133, 2023

    2023

  72. [80]

    X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning

    Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li. X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8563–8573, 2022

    2022

  73. [81]

    Mvt: Multi-view vision transformer for 3d object recognition

    Shuo Chen, Tan Yu, and Ping Li. Mvt: Multi-view vision transformer for 3d object recognition. arXiv preprint arXiv:2110.13083, 2021

    2021 arXiv

  74. [82]

    3dvg-transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937, 2021

    2021

  75. [83]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522–20535, 2022

    2022

  76. [84]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331–7341, 2021

    2021

  77. [85]

    Text-guided graph neural networks for referring 3d instance segmentation

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610–1618, 2021

    2021

  78. [86]

    In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Con...

    2021

  79. [87]

    3d-sps: Single-stage 3d visual grounding via referred point progressive selection

    Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16454–16...

    2022

  80. [88]

    D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X Chang. D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans. 2021

    2021

  81. [89]

    Bottom up top down detection transformers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417–433. Springer, 2022

    2022

  82. [90]

    Ham: Hierarchical attention model with high performance for 3d visual grounding

    Jiaming Chen, Weixin Luo, Xiaolin Wei, Lin Ma, and Wei Zhang. Ham: Hierarchical attention model with high performance for 3d visual grounding. arXiv preprint arXiv:2210.12513, 2(3), 2022

    2022 arXiv

  83. [91]

    3drp-net: 3d relative position-aware network for 3d visual grounding

    Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 3drp-net: 3d relative position-aware network for 3d visual grounding. arXiv preprint arXiv:2307.13363, 2023

    2023 arXiv

  84. [92]

    Eda: Explicit text-decoupling and dense alignment for 3d visual grounding

    Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19231–19242, 2023. 14

    2023

  85. [93]

    the bed, which is rectangular in shape, is located adjacent to the door

    Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. Four ways to improve verbo-visual fusion for dense 3d visual grounding. In European Conference on Computer Vision, pages 196–213. Springer, 2024. 15 A Summary The appendix is organized as follows: Appendix B:More Res...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.