SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Hehe Fan; Xiaoxiao Sun; Yingzhao Jian; Yue Zhang; Yunqiu Xu

REVIEW 4 major objections 6 minor 108 references

SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SmartMage claims that letting the model choose which 3D modalities to use per question, instead of always fusing all of them, is what drives state-of-the-art results on five 3D scene understanding benchmarks.

desk verdict A solid empirical paper on dynamic modality routing for 3D MLLMs, but the headline SOTA numbers hinge on a training-split separation that is never stated; fix that and the results will be credible. read the letter →

arxiv 2608.05137 v1 pith:VYYXNAAP submitted 2026-08-05 cs.CV

Yue Zhang , Yingzhao Jian , Yunqiu Xu , Xiaoxiao Sun , Hehe Fan This is my paper

classification cs.CV

keywords 3Dsceneunderstandingmultimodallargelanguagemodeladaptivemodalityselectionmixtureofexpertsroutingvisualgroundingdensecaptioningquestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SmartMage argues that the main bottleneck in 3D scene understanding is not the number of modalities a model sees, but whether it uses the right modality for each question. The paper proposes a two-stage design: first, a semantic router (SMART) selects task-relevant complementary modalities for each instruction, keeping RGB as the primary input; second, a modality-aware expert gating module (MAGE) routes tokens to specialized experts inside the language model. On five benchmarks covering 3D question answering, dense captioning, and visual grounding, the authors report state-of-the-art results, including gains of +5.1 Acc@0.5 on ScanRefer and +6.4 F1@0.5 on Multi3DRefer over the previous best system. Controlled comparisons show the improvement comes from dynamic orchestration rather than from adding more modalities or switching to a stronger backbone.

What carries the argument

The central mechanism is the pairing of two routing layers: SMART, a global modality scheduler that computes a routing distribution over RGB, depth, BEV, point cloud, and voxel inputs from semantic prior, text-modality similarity, and a quality score derived from feature activation statistics; and MAGE, a local token-level router inside a sparse mixture-of-experts language model that injects a modality-expert affinity prior into expert gating. Together they convert multimodal fusion from a static concatenation into an interpretable, query-dependent pipeline where each question type follows a distinct modality-to-expert pathway.

What would settle it

Re-train SmartMage with evaluation scenes strictly excluded from the unified training corpus and compare against the best fixed-fusion baseline under identical token budgets and backbones; if the gap shrinks to zero or reverses, the claimed benefit of dynamic orchestration is not real.

Watch

Extended reading notes

Core claim

The paper's central claim is that fixed multimodal fusion is structurally wasteful for 3D scene understanding: irrelevant modalities inject semantic noise, while informative ones receive too little attention. SmartMage instead decomposes multimodal reasoning into semantic-guided modality selection and modality-aware expert specialization. The SMART module scores each complementary modality with three signals (instruction-driven semantic prior, text-modality semantic similarity, and activation-based modality quality) and selects a per-query subset, while the MAGE module biases sparse mixture-of-experts routing through a learned modality-expert affinity prior. The authors support the claim with ablations showing adaptive selection outperforms every fixed combination of the same modalities, with the full model reaching 32.6 EM@1 on ScanQA, 66.8 EM@1 on SQA3D, 88.7 CIDEr@0.5 on Scan2Cap, 59.5 Acc@0.5 on ScanRefer, and 60.7 F1@0.5 on Multi3DRefer.

Load-bearing premise

The unified training corpus is built from ScanNet-based annotations, and the paper does not state that the standard training and evaluation scene splits are strictly separated, so if the same scenes or annotations appear on both sides the reported gains could be inflated.

Editorial extensions

If this is right

If the central claim holds, static fusion baselines on 3D benchmarks will need to be re-examined, because adding more modalities can hurt rather than help when routing is fixed.
The diagnostic benchmark ScanFacet implies that modality preference is predictable from semantic category: color and material questions favor appearance cues, while spatial, counting, and shape questions favor geometric cues.
The claimed robustness on RGB-only video benchmarks suggests the router can gracefully degrade when 3D modalities are absent, which matters for real deployments where sensor availability varies.
The reported training-efficiency gain indicates that dynamic routing can reduce per-iteration cost relative to fixed-fusion systems that process all modalities every time.
The observed modality-expert specialization patterns suggest token-level routing inside the language model can be meaningfully aligned with input provenance rather than left fully unsupervised.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the same two-stage design (semantic selection plus modality-conditioned expert routing) could transfer beyond 3D scenes to other heterogeneous modality sets, such as audio-plus-vision or multispectral inputs, because neither module is tied to 3D geometry.
The ScanFacet results implicitly predict a testable ceiling: if each semantic facet were fed only its preferred modality, performance should approach the full SmartMage result; the paper does not run this experiment directly.
The modality-quality evaluator suggests a more general use as a reliability monitor for degraded or missing inputs, which the authors only partially explore through their RGB-only video experiments.
A natural extension the paper leaves implicit is using the learned affinity matrix to prune experts per modality at inference time, which could reduce activated parameters beyond what is reported.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the one-sentence take: this paper is a solid empirical contribution on dynamic modality selection for 3D MLLMs, but the headline SOTA numbers rest on a training-corpus split that the paper never explicitly confirms; that needs to be fixed before the results are fully credible.

What is actually new: the SMART/MAGE decomposition—semantic-guided global routing plus modality-aware expert gating—is a reasonable and non-trivial extension of existing MoE work, and the paper does not oversell its novelty. The controlled comparison in Figure 13b (matching Ross3D on modalities, resolution, backbone) is the right kind of evidence and shows the gains are not just from adding compute or inputs. The ablations in Tables 2–4 are thorough, and the new ScanFacet diagnostic benchmark is a genuinely useful tool for studying modality–semantic patterns.

The soft spots, in rough order of severity:

1. Train/test leakage risk. The paper builds a unified training corpus by merging ScanNet-derived datasets (ScanQA, SQA3D, Scan2Cap, ScanRefer, Multi3DRefer) but never states that only official training splits were used and that evaluation scenes were excluded. All these benchmarks share the same underlying ScanNet scenes, and their official train/test splits differ. If the merged corpus includes scenes that appear in evaluation, the SOTA numbers in Table 1 are inflated by memorization. This affects every benchmark and the central causal claim. The paper does not release code, data manifests, or split-assignment details, so a reviewer cannot verify. This is a real, load-bearing omission, not a nitpick. The limitations section acknowledges token budget and annotation noise but does not mention split separation, which is telling.

2. Missing baseline: the authors cite their own prior Uni3D-MoE but never compare against it. Given the architectural overlap in MoE-based multimodal 3D understanding, a direct comparison is expected. Its absence makes the marginal contribution of SMART+MAGE harder to isolate.

3. No error bars or multiple seeds. Many papers in this area do the same, so this is a moderate concern, not a dealbreaker, but for +1.8 to +6.4 point claims, some variance estimate would help.

4. No code release. The project page is mentioned but no code or data are provided; that limits reproducibility.

Who is this for: researchers working on 3D perception, multimodal LLM routing, and MoE-based reasoning. The diagnostic benchmark and routing analyses are useful independent of the SOTA claims.

Recommendation: this deserves a serious referee. The concerns are addressable—the authors should state split separation explicitly, add the Uni3D-MoE comparison, and ideally release code/data. If the split is clean, the core claim holds and this is a strong paper. If not, the headline numbers fall. I would send it to review with a request for those clarifications.

Referee Report

4 major / 6 minor

Summary. The paper proposes SmartMage, a multimodal large language model for 3D scene understanding that dynamically selects among RGB, depth, BEV, point cloud, and voxel modalities on a per-query basis. A Semantic-guided Modality Adaptive Routing (SMART) module chooses complementary modalities from semantic priors, text-modality alignment, and modality quality, while a Modality-Aware Gating Expert (MAGE) module injects modality-level priors into sparse mixture-of-experts routing. The authors report state-of-the-art results on ScanQA, SQA3D, Scan2Cap, ScanRefer, and Multi3DRefer, competitive RGB-only video results, and a new diagnostic benchmark, ScanFacet, that organizes questions into semantic facets and shows modality preferences per facet. The central claim is that the reported gains come from dynamic modality orchestration rather than from adding more modalities or using a stronger backbone.

Significance. If the empirical claims hold, SmartMage is a useful contribution to query-dependent multimodal fusion for 3D scene understanding, with broad benchmark coverage and a clear architectural decomposition. The paper's strengths include extensive ablations of each routing and gating component (Tables 2-4), a controlled backbone/input comparison in Figure 13, an efficiency analysis (Table 5), a diagnostic benchmark in ScanFacet, and an explicit failure-case analysis in Appendix E.5. No code or data manifest is currently provided, and the central claim is empirical rather than theoretical, so the significance is conditional on the verifiability of the experimental setup and the statistical robustness of the reported gains.

major comments (4)

[§4.1, Appendix A.1] The paper never states that the unified training corpus respects the official train/test splits of ScanNet. All five benchmark evaluations in Table 1 and the ablations in Tables 2-4 are run on models trained on this corpus, and ScanFacet is built from ScanQA/SQA3D pairs. If any annotation from evaluation scenes or a benchmark's own test set enters training, every reported SOTA gain could be inflated by memorization rather than by SMART+MAGE. Please specify exactly which split of each dataset was used for training, confirm that evaluation scenes are excluded, and provide a split manifest or a clear data-release plan; ideally, also report results after retraining on strictly separated official train splits.
[Figure 13(b), §4.3] The controlled comparison with Ross3D is described as 'identical modalities, resolution, and backbone,' but the paper does not state whether Ross3D was retrained on the same unified training corpus, loss terms, optimization schedule, and evaluation protocol, or whether published Ross3D numbers were reused. Since Ross3D's training data may differ from SmartMage's, the performance difference in Figure 13(b) cannot yet be attributed solely to dynamic modality orchestration. Please clarify the exact comparison protocol; if Ross3D was not retrained under identical conditions, the comparison should be labeled as 'same inputs/backbone but different training data,' and the causal attribution should be qualified accordingly.
[Tables 1-4, Figure 13(b)] All reported results appear to come from single training runs, with no error bars, multiple seeds, or statistical significance tests. Several gains are small relative to typical benchmark noise, for example +1.8 EM@1 on ScanQA in Table 1 and the 0.5-1.5 point differences among loss ablations in Table 3. Please report means and standard deviations over at least three seeds, or an equivalent statistical analysis, for the main comparisons and the controlled comparison; without this, the claim that SmartMage is state of the art and that the gains come from routing rather than noise is not fully supported.
[Appendix D.1, Table 11, Figure 5] ScanFacet is constructed from ScanQA and SQA3D question-answer pairs, but the paper does not say whether these pairs come from the training or test splits. The modality-preference analysis in Figure 5 and Table 11 is a descriptive summary of the trained model's behavior; if the same queries were seen during training, the observed 'semantic-modality patterns' do not provide independent evidence of generalizability. Please clarify the split provenance of ScanFacet and, if it is intended as a diagnostic, either use held-out questions or explicitly characterize it as a post-hoc analysis of the trained model rather than an evaluation benchmark.

minor comments (6)

[Table 14] The row labeled 'Chat-Scene [39]' does not match the reference list: reference [39] is a survey by Li et al., while Chat-Scene is reference [29]. Please correct the citation and verify all table-reference correspondences.
[Appendix E.2, Table 16] The expert-scaling study changes both the number of experts and the per-expert hidden dimension (11008→1408) simultaneously, so the performance trend cannot be attributed to expert count alone despite the caption's claim of a 'cleaner analysis.' Please separate the two factors or clearly state that the result is a joint effect of expert count and per-expert capacity.
[Table 5] The column header 'Train All Params.' is ambiguous; it is unclear whether the slash-separated numbers denote trainable versus total parameters, backbone versus MoE parameters, or some other split. The caption should define the notation for both training and inference cost columns.
[Figure 1] The label 'Brid-eye-view Map' contains a typo; it should be 'Bird's-Eye-View Map.'
[Table 12] Ross3D is listed twice in the 3D MLLMs section of Table 12 with identical numbers; please deduplicate the entry so the reference result is reported once.
[Reproducibility] Because the paper's empirical claims depend on the exact composition of the unified training corpus, please state on the project page whether code and a split manifest will be released; the current text provides only a project page URL without a data or code availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SmartMage's claims are empirical comparisons on external benchmarks, not derivations from fitted inputs or self-citations.

full rationale

The paper makes no mathematical derivation whose output is equivalent to its inputs. The central SOTA claims are obtained by training SmartMage on a unified corpus of ScanNet-based annotations and evaluating on standard public benchmarks (ScanQA, SQA3D, Scan2Cap, ScanRefer, Multi3DRefer), with comparisons to external baselines. The SMART/MAGE modules are trained with supervised losses, and the ablations show incremental gains; none of these results is a fitted parameter renamed as a prediction. The ScanFacet benchmark is a reorganization of existing QA pairs into semantic categories and is used descriptively, not as a training target; its modality-preference plots are observational. The only self-citations (e.g., Uni3D-MoE [88]) appear in contextual reference lists and are not load-bearing. The unstated train/test split separation for the unified corpus is a potential validity risk, but it is not a circularity pattern under the defined criteria: no equation or definition makes the evaluation equal to the training input. Therefore score 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about how well semantic cues, activation statistics, and geometric coverage reflect usefulness. These are not invalid, but they are unverified choices that could affect routing decisions.

free parameters (7)

lambda_dis = 1.0
Loss weight for discrimination loss in SMART; chosen by hand, no sensitivity analysis.
lambda_sem = 0.5
Loss weight for semantic correlation loss; chosen by hand.
lambda_bal = 0.01
Loss weight for expert balancing loss; chosen by hand.
gating temperature = 1.0
Temperature in MoE routing and expert prior, set in Appendix B.3.
routing weights alpha_s, alpha_p, alpha_q = 1.0, 0.7, 0.5
Weights for semantic similarity, prior, and quality in Eq. (15).
selection threshold rho = 0.8
Cumulative probability threshold for adaptive modality selection in Appendix B.2.
k_max = 3
Maximum number of complementary modalities selected per query.

assumptions (5)

domain assumption Textual instructions indicate which modalities are most informative for answering.
The SMART module's semantic prior estimator is built on this assumption, e.g., 'What color' implies RGB.
ad hoc to paper Activation statistics (strength, sparsity, stability) of modality features are a faithful proxy for modality quality.
Modality Quality Evaluator in Section 3.2 and Figure 3 uses these statistics without independent justification.
domain assumption Voxel coverage computed from camera poses is a good proxy for view informativeness in keyframe selection.
FoVSR algorithm in Appendix A.3.1 prunes voxels and selects frames by coverage.
domain assumption The unified training corpus respects standard ScanNet train/test splits, so no evaluation leakage occurs.
Section 4.1 states the corpus is built from ScanNet annotations but does not specify split separation.
domain assumption Mask3D proposals are of sufficient quality for fair visual grounding evaluation.
The proposal-based grounding protocol in Appendix C.3 depends on Mask3D detections.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding." pith.science (2026). https://pith.science/paper/VYYXNAAP

@misc{pith2026260805137,
  author       = {Pith},
  title        = {Pith review of: SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VYYXNAAP}},
  note         = {Machine review of arXiv:2608.05137}
}

read the original abstract

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.

Figures

Figures reproduced from arXiv: 2608.05137 by the authors.

**Figure 1.** Motivation of SmartMage. (a) Different question types exhibit inherent modality preferences. (b) Previous MLLMs fuse [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Overview of SmartMage. The omni-modal feature extractor first encodes text, RGB, depth, BEV, point cloud, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Details of the Modality Quality Evaluator. We [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Overview of the end-to-end training strategy. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 6.** Figure 6: Modality-aware expert speculation. Comparison of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Hierarchical routing from question types to modal [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Qualitative visualization. The model adaptively selects task-relevant modalities for interpretable reasoning. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 11.** Figure 11: Prompt templates for different tasks. Unified [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

**Figure 10.** Figure 10: Prompt template. A system message defines the [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

**Figure 12.** Figure 12: Examples of 3D scene understanding tasks, including 3D visual grounding (ScanRefer [ [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: Controlled comparison of SmartMage. (a) Smart [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Visualization of SmartMage performing diverse 3D scene understanding tasks on ScanFacet. Examples highlight the [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

**Figure 15.** Figure 15: Visualization of modality-specific routing trajectories across MoE layers. The left column shows the top-10 activated [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

**Figure 16.** Figure 16: Failure cases of SmartMage. Top: an annotation-ambiguity example where the model selects object 13 for the [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

108 extracted references · 58 canonical work pages

[1]

Mahmoud Ahmed, Junjie Fei, Jian Ding, Eslam Mohamed Bakr, and Mohamed Elhoseiny. 2025. Kestrel: 3D multimodal LLM for part-aware grounded description. InICCV

2025
[2]

Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. 2025. Llava-onevision- 1.5: Fully open framework for democratized multimodal training.arXiv preprint arXiv:2509.23661(2025)

arXiv 2025
[3]

Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. Scanqa: 3d question answering for spatial scene understanding. InCVPR

2022
[4]

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al . 2025. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631(2025)

arXiv 2025
[5]

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. InNeurIPS

2022
[6]

Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 2022. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. InCVPR

2022
[7]

Rui Cai, Bangzheng Li, Xiaofei Wen, Muhao Chen, and Zhe Zhao. 2025. Diagnos- ing and mitigating modality interference in multimodal large language models. arXiv preprint arXiv:2505.19616(2025)

arXiv 2025
[8]

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang

Show all 108 references

[9]

Qihang Cao and Huangxun Chen. 2025. Objvariantensemble: Advancing point cloud llm evaluation in challenging scenes with subtly distinguished objects. In AAAI

2025
[10]

Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. InECCV

2020
[11]

Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X Chang. 2022. D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans. InECCV

2022
[12]

Jiaming Chen, Weixin Luo, Xiaolin Wei, Lin Ma, and Wei Zhang. 2022. Ham: Hierarchical attention model with high performance for 3d visual grounding. arXiv preprint arXiv:2210.12513(2022)

2022 arXiv
[13]

Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. 2024. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InCVPR

2024
[14]

Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Language conditioned spatial relation reasoning for 3d object grounding. InNeurIPS

2022
[15]

Shuhang Chen, Yunqiu Xu, Junjie Xie, Aojun Lu, Tao Feng, Zeying Huang, Zhang Ning, Yi Sun, Yi Yang, and Hangjie Yuan. 2026. Cogflow: Bridging perception and reasoning through knowledge internalization for visual mathematical problem solving. InICLR

2026
[16]

Shuo Chen, Tan Yu, and Ping Li. 2021. Mvt: Multi-view vision transformer for 3d object recognition. InBMVC

2021
[17]

Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. 2023. End-to-end 3d dense captioning with vote2cap-detr. InCVPR

2023
[18]

Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. 2024. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370(2024)

2024 arXiv
[19]

Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. 2021. Scan2cap: Context-aware dense captioning in rgb-d scans. InCVPR

2021
[20]

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InCVPR

2017
[21]

Xianzhi Du, Tom Gunter, Xiang Kong, Mark Lee, Zirui Wang, Aonan Zhang, Nan Du, and Ruoming Pang. 2024. Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training.arXiv preprint arXiv:2405.15052(2024)

2024 arXiv
[22]

Hehe Fan, Yi Yang, and Mohan Kankanhalli. 2022. Point spatio-temporal trans- former networks for point cloud video modeling.TPAMI(2022)

2022
[23]

Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. 2025. Scene-LLM: Extending Language Model for 3D Visual Reasoning. InW ACV

2025
[24]

Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. 2024. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. InICRA

2024
[25]

Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. 2023. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following.arXiv preprin...

2023 arXiv
[26]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531(2015)

2015 arXiv
[27]

Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3d-llm: Injecting the 3d world into large language models. InNeurIPS

2023
[28]

Wenjin Hou, Wei Liu, Han Hu, Xiaoxiao Sun, Serena Yeung-Levy, and Hehe Fan
[29]

Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al . 2024. Chat-scene: Bridging 3d scene and large language models with object identifiers. InNeurIPS

2024
[30]

Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. 2023. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.arXiv preprint arXiv:2312.08168(2023)

2023 arXiv
[31]

Junzhe Huang, Xiaoxiao Sun, Yan Yang, Yuxuan Hou, Ruotian Zhang, Sirui Li, Hehe Fan, Serena Yeung-Levy, and Xin Yu. 2026. WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild.arXiv preprint arXiv:2605.01018(2026)

2026 arXiv
[32]

Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2024. An Embodied Generalist Agent in 3D World. InICML. MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Zhang et al

2024
[33]

Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. 2021. Text-guided graph neural networks for referring 3d instance segmentation. In AAAI

2021
[34]

Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki
[35]

Heng Jia, Yunqiu Xu, Linchao Zhu, Guang Chen, Yufei Wang, and Yi Yang. 2024. MoS2: Mixture of scale and shift experts for text-only video captioning. InACM MM

2024
[36]

Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. 2023. Context- aware alignment and mutual masking for 3d-language pre-training. InCVPR

2023
[37]

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. InCVPR

2021
[38]

Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. 2025. Uni-moe: Scaling unified multimodal llms with mixture of experts.TPAMI(2025)

2025
[39]

Zechuan Li, Hongshan Yu, Yihao Ding, Yan Li, Yong He, and Naveed Akhtar. 2025. Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering.Information Fusion(2025)

2025
[40]

Hanwen Liang, Dejia Xu, Neel P Bhatt, Hezhen Hu, Hanxue Liang, and Kon- stantinos N Plataniotis. 2026. Comp4D: Compositional 4D Scene Generation. In W ACV

2026
[41]

Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, and Li Yuan. 2026. Moe-llava: Mixture of experts for large vision-language models.TMM(2026)

2026
[42]

Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yong- shun Gong, Peng Gao, and Wanli Ouyang. 2024. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models.arXiv preprint arXiv:2402.03327(2024)

2024 arXiv
[43]

Yuetong Liu, Yunqiu Xu, Yang Wei, Xiuli Bi, and Bin Xiao. 2026. Clear nights ahead: Towards multi-weather nighttime image restoration. InAAAI

2026
[44]

Dongchen Lu, Yuyao Sun, Zilu Zhang, Leping Huang, Jianliang Zeng, Mao Shu, and Huo Cao. 2025. InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression.arXiv preprint arXiv:2503.21307(2025)

2025 arXiv
[45]

Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 2022. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. InCVPR

2022
[46]

Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2023. Sqa3d: Situated question answering in 3d scenes. In ICLR

2023
[47]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748(2018)

2018 arXiv
[48]

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. InNeurIPS

2017
[49]

Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. 2024. Gpt4point: A unified framework for point-language understanding and generation. InCVPR

2024
[50]

Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao
[51]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. InCVPR

2015
[52]

Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. 2023. Mask3d: Mask transformer for 3d semantic instance segmentation. InICRA

2023
[53]

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InICLR

2017
[54]

GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models. InICLR
[55]

Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. 2024. Promoe: Fast moe-based llm serving using proactive caching.arXiv preprint arXiv:2410.22134 (2024)

2024 arXiv
[56]

Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, and Marc Pollefeys. 2024. Space3D-Bench: Spatial 3D question answering benchmark. arXiv preprint arXiv:2408.16662(2024)

2024 arXiv
[57]

Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. 2024. Four ways to improve verbo-visual fusion for dense 3d visual grounding. InECCV

2024
[58]

Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. 2023. Scaling vision-language models with sparse mixture of experts. In Findings of EMNLP

2023
[59]

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

2024 arXiv
[60]

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al
[61]

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. 2025. Internvl3. 5: Ad- vancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265(2025)

2025 arXiv
[62]

Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. 2025. ROSS3D: Reconstructive Visual Instruction Tuning with 3D-Awareness. InCVPR

2025
[63]

Yuan Wang, Ya-Li Li, WU Eastman ZY, and Shengjin Wang. 2025. LIBA: Language Instructed Multi-granularity Bridge Assistant for 3D Visual Grounding. InAAAI

2025
[64]

Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 2023. 3drp-net: 3d relative position-aware network for 3d visual grounding. InEMNLP

2023
[65]

Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, Tao Jin, and Zhou Zhao
[66]

Shicai Wei, Chunbo Luo, and Yang Luo. 2025. Improving multimodal learning via imbalanced learning. InICCV

2025
[67]

Xiaoyan Wang, Zeju Li, Yifan Xu, Jiaxing Qi, Zhifei Yang, Ruifei Ma, Xiangde Liu, and Chao Zhang. 2025. Spatial 3d-llm: Exploring spatial awareness in 3d vision-language models. InICME

2025
[68]

Chen Henry Wu, Neil Kale, and Aditi Raghunathan. 2025. Mitigating modal imbalance in multimodal reasoning. InCOLM

2025
[69]

Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. 2023. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In CVPR

2023
[70]

Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, and Huchuan Lu. 2025. 3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding.TMM(2025)

2025
[71]

Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2024. Pointllm: Empowering large language models to understand point clouds. InECCV

2024
[72]

Yunqiu Xu, Linchao Zhu, and Yi Yang. 2024. Gg-editor: Locally editing 3d avatars with multimodal large language model guidance. InACM MM

2024
[73]

Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. 2024. Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation. InICRA

2024
[74]

Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. InICML

2024
[75]

Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. 2024. Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. InICRA

2024
[76]

Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie
[77]

Senqiao Yang, Jiaming Liu, Renrui Zhang, Mingjie Pan, Ziyu Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Hongsheng Li, Yandong Guo, et al. 2025. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. InAAAI

2025
[78]

Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. 2026. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence. InICLR

2026
[79]

Yunqiu Xu, Linchao Zhu, and Yi Yang. 2025. Mc-bench: A benchmark for multi- context visual grounding in the era of mllms. InICCV

2025
[80]

Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. 2024. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. InNeurIPS

2024
[81]

Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. 2023. Lamm: Language- assisted multi-modal instruction-tuning dataset, framework, and benchmark. In NeurIPS

2023
[82]

Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li. 2022. X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. InCVPR

2022
[83]

Thinking in space: How multimodal large language models see, remember, and recall spaces. InCVPR
[84]

Tongtian Yue, Longteng Guo, Jie Cheng, Xuange Gao, Hua Huang, and Jing Liu
[85]

Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. 2025. How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM. InIJCAI

2025
[86]

Shusheng Yang, Jihan Yang, Pinzhi Huang, Ellis Brown, Zihao Yang, Yue Yu, Shengbang Tong, Zihan Zheng, Yifan Xu, Muhan Wang, et al. 2025. Cambrian-s: Towards spatial supersensing in video.arXiv preprint arXiv:2511.04670(2025)

2025 arXiv
[87]

Yiming Zhang, ZeMing Gong, and Angel X Chang. 2023. Multi3drefer: Grounding text description to multiple 3d objects. InICCV. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil

2023
[88]

Yue Zhang, Yingzhao Jian, Hehe Fan, Yi Yang, and Roger Zimmermann. 2025. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts. arXiv preprint arXiv:2505.21079(2025)

2025 arXiv
[89]

Yue Zhang, Chao Wang, Fei Fang, Yunzhi Zhuge, Hehe Fan, Xiaojun Chang, Cheng Deng, and Yi Yang. 2025. SAMControl: Controlling pose and object for image editing with soft attention mask.ToMM(2025)

2025
[90]

Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. 2021. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In ICCV

2021
[91]

Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 2021. 3dvg-transformer: Relation modeling for visual grounding on point clouds. InICCV

2021
[92]

Ada-k routing: Boosting the efficiency of moe-based llms. InICLR
[93]

Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, An- drew M Dai, Quoc V Le, James Laudon, et al . 2022. Mixture-of-experts with expert choice routing. InNeurIPS

2022
[94]

Yue Zhang, Hehe Fan, Wei Ji, Yongkang Wong, Roger Zimmermann, and Yi Yang
[95]

Prompt-aware adapter: Learning adaptive visual tokens for multimodal large language models.TAI(2025)

2025
[96]

Wenxuan Zhu, Bing Li, Cheng Zheng, Jinjie Mai, Jun Chen, Letian Jiang, Abdullah Hamdi, Sara Rojas Martinez, Chia-Wen Lin, Mohamed Elhoseiny, et al . 2025. 4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding. InICCV

2025
[97]

Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 2023. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In ICCV

2023
[98]

Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. 2024. Unifying 3d vision-language understanding via promptable queries. InECCV

2024
[99]

Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun MA, Ziwei Liu, and Chunyuan Li. 2025. LLaVA-Video: Video Instruction Tuning With Synthetic Data.TMLR (2025)

2025
[101]

Duo Zheng, Shijia Huang, and Liwei Wang. 2025. Video-3d llm: Learning position- aware video representation for 3d scene understanding. InCVPR

2025
[103]

Zhenglin Zhou, Fan Ma, Xiaobo Xia, Hehe Fan, Yi Yang, and Tat-Seng Chua
[104]

ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models.arXiv preprint arXiv:2511.22456(2025)

2025
[105]

Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. 2025. Llava-3d: A simple yet effective pathway to empowering lmms with 3d capabili- ties. InICCV

2025
[2022]

Bottom up top down detection transformers for language grounding in images and point clouds. InECCV
[2023]

Image as a foreign language: Beit pretraining for vision and vision-language tasks. InCVPR
[2024]

A survey on mixture of experts.arXiv preprint arXiv:2407.06204(2024)

2024 arXiv
[2025]

InFindings of NAACL

Data-Efficiently Learn Large Language Model for Universal 3D Scene Perception. InFindings of NAACL
[2026]

Seeing is believing? a benchmark for multimodal large language models on visual illusions and anomalies.arXiv preprint arXiv:2602.01816(2026)

2026

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Mahmoud Ahmed, Junjie Fei, Jian Ding, Eslam Mohamed Bakr, and Mohamed Elhoseiny. 2025. Kestrel: 3D multimodal LLM for part-aware grounded description. InICCV

2025

[2] [2]

Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. 2025. Llava-onevision- 1.5: Fully open framework for democratized multimodal training.arXiv preprint arXiv:2509.23661(2025)

arXiv 2025

[3] [3]

Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. Scanqa: 3d question answering for spatial scene understanding. InCVPR

2022

[4] [4]

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al . 2025. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631(2025)

arXiv 2025

[5] [5]

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. InNeurIPS

2022

[6] [6]

Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 2022. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. InCVPR

2022

[7] [7]

Rui Cai, Bangzheng Li, Xiaofei Wen, Muhao Chen, and Zhe Zhao. 2025. Diagnos- ing and mitigating modality interference in multimodal large language models. arXiv preprint arXiv:2505.19616(2025)

arXiv 2025

[8] [8]

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang