Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing 2D multi-view images with camera poses and 3D point-cloud features improves frozen-LLM 3D scene understanding by compensating for lossy point-cloud reconstruction.

desk verdict Solid architecture, but the central ablation for multi-view benefit is an inference-time removal of a jointly trained branch, so the headline claim needs a cleaner test. read the letter →

arxiv 2507.12916 v1 pith:A6TWTOU6 submitted 2025-07-17 cs.CV cs.AI

Yifan Xu , Chao Zhang , Hanqi Jiang , Xiaoyan Wang , Ruifei Ma , Yiwei Li , Zihao Wu , Zeju Li
show 1 more author
Xiangde Liu
This is my paper
classification cs.CVcs.AI
keywords 3Dsceneunderstandingmulti-viewimageslargelanguagemodelsvisualquestionansweringgroundingcameraposespointcloudsmultimodalfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Argus, a 3D large multimodal framework that feeds a frozen large language model both 3D point-cloud features and 2D multi-view images with camera poses. Its central claim is that reconstructed indoor point clouds lose information in textureless or repetitive regions and on complex objects, and that 2D multi-view images, which are visually consistent with the point cloud, can supply that missing detail and improve 3D understanding. This matters because it offers a path to better 3D question answering, visual grounding, captioning, and embodied dialogue without fine-tuning the language model and without collecting more 3D annotations. On ScanQA validation, the full 2D+3D model with a FlanT5-3B backbone reaches 21.8 exact-match and 76.9 CIDEr, compared with 20.5 and 69.4 for the 3D-only 3D-LLM baseline with the same backbone.

What carries the argument

Two trainable modules carry the argument. First, the fusion module uses a Q-Former (a transformer with learnable queries that extract text-relevant visual features) to obtain per-view features, adds position embeddings computed by an MLP from each view's camera pose, and aggregates all views with four stacked transformer layers into view-as-scene features. Second, the 3D-aware Q-Former lets 3D learnable queries first self-attend to the view-as-scene features, then cross-attend to the 3D point-cloud features, and projects the result into the LLM's embedding space. This two-step interaction is what joins the holistic scene structure of the point cloud with the detailed appearance from the images.

What would settle it

Retrain the 3D-only branch of Argus and the leading 3D-LMM baselines under identical training data, backbone, and prompt protocol on ScanQA, then compare against the 2D+3D model; if the multi-view fusion no longer yields a clear margin, the claim that 2D images improve 3D understanding collapses.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that 2D multi-view images, when fused with camera poses and then interacted with 3D point-cloud features through a 3D-aware Q-Former, can effectively improve 3D scene understanding for a frozen LLM. The improved features compensate for voids and distortions that appear when indoor scenes are reconstructed as point clouds. The evidence is the Section IV-F ablation: the 2D+3D model outperforms the 3D-only variant (same 3D-aware Q-Former) and the 2D-only variant, and it beats the 3D-LLM baseline with the same FlanT5-3B backbone across EM, Meteor, Rouge-L, and CIDEr. The paper also shows that the fusion module and the 3D-aware Q-Former each work alone and mutually enhance each other.

Load-bearing premise

The central claim rests on two premises: that the 2D views actually carry scene detail missing from the reconstructed point cloud, and that the published baseline scores are comparable to Argus's even though backbones, training data, and prompt protocols differ.

Editorial extensions

If this is right

  • A frozen-LLM 3D system can improve on ScanQA without more 3D annotation by adding 100 RGB views per scene: CIDEr rises from 69.4 (3D-LLM, FlanT5-3B) to 76.9.
  • The 3D-aware Q-Former can run in 3D-only mode and still beat 3D-LLM (EM 21.1 vs 20.5, CIDEr 72.5 vs 69.4), so the benefit is not purely from adding a second modality.
  • The fusion module can run in 2D-only mode and beat single-image and multi-view 2D baselines by more than 5 CIDEr, showing that pose-conditioned view aggregation is itself useful.
  • Camera poses contribute about +2 CIDEr, so spatial conditioning of views is a real factor rather than just added image content.
  • The framework keeps the LLM frozen across all training stages, which lowers memory and time requirements compared with methods that fine-tune the language backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the gains are tied to fusing many views rather than a single image, the same pose-conditioned fusion module could be evaluated on 2D-only visual question answering to separate the contribution of view aggregation from the contribution of 3D structure.
  • Inference: a natural stress test is sampling the 100 views adversarially, for example all from one corner of the room, to see whether the view-as-scene bottleneck still compensates for point-cloud voids; the paper does not vary view distribution.
  • Inference: if the compensation mechanism is general, Argus-style fusion should transfer to outdoor scenes or RGB-D scans where textureless regions are common, but the paper only evaluates indoor ScanNet scenes, so outdoor transfer is untested.
  • Inference: the ablation showing camera poses add about +2 CIDEr suggests that richer geometric conditioning, such as per-pixel ray encodings or epipolar constraints, could extend the gains beyond the paper's MLP pose embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Argus, a 3D large multimodal foundation model that augments 3D point-cloud features with 2D multi-view images and camera poses for frozen-LLM scene understanding. A fusion module, consisting of a 2D Q-Former, a pose MLP, and four transformer layers, produces view-as-scene features; a 3D-aware Q-Former then lets learnable queries attend to these features and to 3D features, outputting 3D-aware embeddings projected into the LLM. Training is performed in three stages with the LLM kept frozen, using ScanNet annotations from 3D-LLM and 3DMIT. The paper evaluates Argus on ScanQA question answering, ScanRefer 3D visual grounding, embodied dialogue, scene description, and embodied planning, and reports ablations on the number of views, fusion design, training stages, 3D features, camera poses, and Q-Former initialization. The central claim is that fusing multi-view images with 3D point clouds improves 3D scene understanding over 2D-only, 3D-only, and existing 3D-LMM baselines.

Significance. If the central claim were firmly established, Argus would be a useful empirical contribution: it avoids LLM fine-tuning, uses a relatively modest 75k annotation budget, reuses strong off-the-shelf components (EVA-CLIP, EPCL, BLIP-2 Q-Former), and offers flexible 2D-only, 3D-only, and 2D+3D inference. The system is evaluated across multiple downstream tasks and includes thoughtful ablations (view count, fusion aggregation, training stages, features, poses, initialization), and the training-resource comparison in Table II is informative. However, the evidence for the headline superiority claim is not yet commensurate with the text: the key modality ablation is confounded by train/inference mismatch, and the external baselines are not matched in backbone, data, or protocol.

major comments (4)
  1. [Section IV-F, Table V] The 'Argus 3D' and 'Argus 2D' rows are not clean modality ablations. As Fig. 4 and the text state, these rows are obtained by separating the fusion module and the 3D-aware Q-Former from a fine-tuned model that was trained jointly on 2D+3D input. In training, the 3D-aware Q-Former's self-attention always had view-as-scene features as context, and the LLM always received 3D-aware embeddings; at 3D-only inference the view-as-scene features are absent, while at 2D-only inference the LLM receives raw view-as-scene features it never saw during training. The 2D+3D advantage in Table V (21.8 vs 21.1 EM; 76.9 vs 72.5 CIDEr) could therefore be an artifact of train/test input mismatch. To support the claim that multi-view images improve 3D scene understanding, the authors should train a genuine 3D-only model (and a genuine 2D-only model) under the same protocol and compare.
  2. [Abstract, Section IV-C, Table I] The statement that Argus 'outperforms existing 3D-LMMs' is contradicted by the numbers in Table I for NaviLLM, which achieves 23.0 EM on ScanQA validation and 26.3 EM on test versus Argus's 21.8 and 25.9. Moreover, the baseline comparisons are not controlled: 3D-LLM uses FlanT5-3B, Opt-1.3B, or Flamingo-3B; 3DMIT and NaviLLM use Vicuna-7B; and training data and prompt protocols differ across systems. NaviLLM additionally trains on Matterport3D navigation data, which Section IV-C acknowledges may contribute to its performance. A claim of superiority requires either a properly matched re-implementation of baselines under Argus's training protocol or a clear statement of the settings in which Argus is and is not superior.
  3. [Section IV, especially Tables I and V] No multiple runs, standard deviations, or significance tests are reported anywhere, and several headline differences are small (e.g., 0.7 EM on ScanQA validation; 1.0 EM between 2D+3D and 3D-only in Table V). Given that the central conclusion rests on these margins, the authors should report seed variance or significance tests for the key comparisons.
  4. [Section IV-F and Table XI] The claim that 'when 2D multi-view images are not integrated, our method simplifies to 3D-LLM' is inaccurate because the 3D-aware Q-Former is a different architecture and is initialized from the 2D BLIP-2 Q-Former. Table XI shows that this initialization alone improves EM from 20.3 to 21.4 and CIDEr from 71.2 to 74.5 on 3D-LLM features. This confounds the Argus-3D versus 3D-LLM comparison in Table V and should be disentangled from the effect of multi-view input.
minor comments (6)
  1. [Section IV-F, Table V] The SingleImage and MultiView baselines are insufficiently specified: it is not clear whether they use the fusion module, how multi-view features are concatenated or pooled, or what training data and instruction format were used; please provide full implementation details.
  2. [Section IV-G, Tables V and VI] The number-of-views ablation in Table VI reports 21.4 EM for 100 views, while Table V reports 21.8 EM for the 2D+3D configuration that uses 100 views; please state which training-stage setting each table uses and reconcile the discrepancy.
  3. [Section IV-E, Table IV] EM scores are reported only for Argus in Table IV; please clarify whether EM was computed for LL3DA and the zero-shot baselines and why those entries are missing.
  4. [Fig. 5] The green arrows labeled '+multi-view images' in Fig. 5 are not explained in the caption; please clarify how the arrows relate to the failure cases.
  5. [Throughout] The title uses '3-D' while the main text uses '3D', and the acronym '3D-LMM' appears in the abstract without expansion; please standardize the terminology and define the acronym at first use.
  6. [Reproducibility] No code or model release is mentioned; releasing the trained checkpoints and training scripts would materially aid reproducibility, given the many moving parts (LAVIS initialization, EPCL, 3D-LLM features).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central 2D+3D fusion claim is tested against external baselines and ablations; the only self-citation (3DMIT) is not load-bearing.

full rationale

The paper's derivation chain is not circular in the sense required by the analysis. The core architecture combines externally attributed components (BLIP-2 Q-Former, EVA-CLIP, EPCL, 3D-LLM features) and is evaluated on ScanQA, ScanRefer, and additional tasks against published baselines, so the headline results are not defined in terms of the paper's own fitted parameters. The main self-citation is the authors' prior 3DMIT data and baseline, which serve as pretraining material and a comparison point; this does not force the outcome and is not the sole support for the method's claims. The strongest concern is Section IV-F, where the 'Argus 3D-only' and 'Argus 2D-only' rows in Table V are obtained by separating the fusion module and 3D-aware Q-Former from the jointly trained model, so the measured gain of 2D+3D over these modes may reflect train/inference input-distribution mismatch rather than a clean modality benefit. However, this is an experimental-design confound that weakens the internal ablation, not a circular definition, a fitted parameter renamed as a prediction, or a result that reduces by construction to its own inputs. The paper also explicitly acknowledges spatial-relation limitations in Section IV-H, and the external baseline comparisons provide independent content. A score of 1 reflects the minor self-citation without treating it as load-bearing circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical ML paper: there is no derivation of a physical constant. The ledger records the domain assumptions the architecture and evaluation rely on, plus the hyperparameters selected by validation. No new physical entities are introduced.

free parameters (3)
  • Number of multi-view images per scene = 100
    Selected by ablation over {80, 100, 120}; Table VI shows only small differences, and 100 is used for all main experiments.
  • Fusion transformer depth = 4 stacked transformer layers
    Chosen by design; no ablation is reported for this architectural choice.
  • 3D learnable query count = 32
    Inherited from Q-Former convention; stated to affect representative richness and computational overhead.
assumptions (4)
  • domain assumption Multi-view 2D images and 3D point cloud features are semantically consistent enough for cross-attention transfer to be useful.
    Assumed throughout Section III-D; the 3D-aware Q-Former's cross-attention relies on this without a formal guarantee.
  • domain assumption Pretrained EVA-CLIP ViT-g/14 and 3D feature extractors (EPCL or 3D-LLM features) produce text-aligned features.
    Relied on in Sections III-B and III-C; no alignment verification is provided.
  • domain assumption Published baseline numbers in Tables I, III, and IV are directly comparable to Argus's numbers.
    The superiority claim depends on cross-paper comparison with different LLM backbones and training data; the paper itself notes NaviLLM's extra navigation data.
  • domain assumption ScanNet scenes and language annotations are representative of 3D scene understanding tasks.
    The paper discusses indoor bias in Section IV-H, acknowledging limited generalization to outdoor scenes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models." pith.science (2026). https://pith.science/paper/A6TWTOU6

@misc{pith2026250712916,
  author       = {Pith},
  title        = {Pith review of: Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6TWTOU6}},
  note         = {Machine review of arXiv:2507.12916}
}
read the original abstract

Advancements in foundation models have made it possible to conduct applications in various downstream tasks. Especially, the new era has witnessed a remarkable capability to extend Large Language Models (LLMs) for tackling tasks of 3D scene understanding. Current methods rely heavily on 3D point clouds, but the 3D point cloud reconstruction of an indoor scene often results in information loss. Some textureless planes or repetitive patterns are prone to omission and manifest as voids within the reconstructed 3D point clouds. Besides, objects with complex structures tend to introduce distortion of details caused by misalignments between the captured images and the dense reconstructed point clouds. 2D multi-view images present visual consistency with 3D point clouds and provide more detailed representations of scene components, which can naturally compensate for these deficiencies. Based on these insights, we propose Argus, a novel 3D multimodal framework that leverages multi-view images for enhanced 3D scene understanding with LLMs. In general, Argus can be treated as a 3D Large Multimodal Foundation Model (3D-LMM) since it takes various modalities as input(text instructions, 2D multi-view images, and 3D point clouds) and expands the capability of LLMs to tackle 3D tasks. Argus involves fusing and integrating multi-view images and camera poses into view-as-scene features, which interact with the 3D features to create comprehensive and detailed 3D-aware scene embeddings. Our approach compensates for the information loss while reconstructing 3D point clouds and helps LLMs better understand the 3D world. Extensive experiments demonstrate that our method outperforms existing 3D-LMMs in various downstream tasks.

Figures

Figures reproduced from arXiv: 2507.12916 by the authors.

Figure 1
Figure 1. High-level overview of our proposed method Argus. It receives 2D multi-view images and 3D point cloud input and generates responses to text instructions. Argus can generalize to various downstream tasks in 3D scenes. patterns (such as uniform walls, homogeneous floor tiles, and glass surfaces) lack distinctive features within the captured images. This deficiency significantly hampers the efficacy of stereo-matching … view at source ↗
Figure 2
Figure 2. Overall pipeline of our method. The fusion module aggregates multi-view images and their corresponding camera poses into view-as-scene features. The learnable queries of 3D-aware Q-Former interact with the 3D features and view-as-scene features. The outputs of 3D-aware Q-Former(3D-aware embeddings) are then projected to the embedding space of LLM. and boxes. 3DMIT [26] proposes an efficient tuning method for 3D-LLMs… view at source ↗
Figure 3
Figure 3. Structure of the fusion module. The module involves two key steps: extracting multi-view features with spatial information and aggregating these features as view-as-scene features. By capturing detailed information about the scene, the view-as-scene features facilitate the acquisition of comprehensive 3D-aware embeddings. Specifically, the 3D-aware Q-Former incorporates a set of trainable vectors, 3D Learnable Query… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Three inference structures of Argus. After training with 2D multi-view [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Qualitative Results. We provide several visualization results of various tasks in diverse 3D environments, including 3D-QA, embodied dialogue, scene description, embodied planing and 3D-VG. Failure cases are visualized in red. The green arrow denotes leveraging 2D mult…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Spatial 3D-LLM adds a progressive spatial awareness scheme to a 3D vision-language model, improving several 3D understanding and grounding metrics and introducing new distance and layout-editing tasks.

Reference graph

Works this paper leans on

61 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. Mach. Learn. PMLR, 2021, pp. 8748–8763

    work page 2021

  2. [2]

    Uni3d: Exploring unified 3d representation at scale,

    J. Zhou, J. Wang, B. Ma, Y .-S. Liu, T. Huang, and X. Wang, “Uni3d: Exploring unified 3d representation at scale,” in Proc. Int. Conf. Learn. Represent., 2024

    work page 2024

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 13

    arXiv 2023

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” 2023

    2023

  6. [6]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma et al., “Scaling instruction-finetuned language models,” J. Mach. Learn. Res., vol. 25, no. 70, pp. 1–53, 2024

    work page 2024

  7. [7]

    Improving pretrained language model fine-tuning with noise stability regularization,

    H. Hua, X. Li, D. Dou, C.-Z. Xu, and J. Luo, “Improving pretrained language model fine-tuning with noise stability regularization,” IEEE Trans. Neural Netw. & Learn. Syst. , 2023

    work page 2023

  8. [8]

    Topicbert: A topic- enhanced neural language model fine-tuned for sentiment classification,

    Y . Zhou, L. Liao, Y . Gao, R. Wang, and H. Huang, “Topicbert: A topic- enhanced neural language model fine-tuned for sentiment classification,” IEEE Trans. Neural Netw. & Learn. Syst. , 2021

    work page 2021

Show all 61 references
  1. [9]

    Flamingo: a visual lan- guage model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual lan- guage model for few-shot learning,” arXiv preprint arXiv:2204.14198 , 2022

    2022 arXiv

  2. [10]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  3. [11]

    Multimodal-gpt: A vision and language model for dialogue with humans,

    T. Gong, C. Lyu, S. Zhang, Y . Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen, “Multimodal-gpt: A vision and language model for dialogue with humans,” arXiv preprint arXiv:2305.04790 , 2023

    2023 arXiv

  4. [12]

    Octavius: Mitigating task interference in mllms via moe,

    Z. Chen, Z. Wang, Z. Wang, H. Liu, Z. Yin, S. Liu, L. Sheng, W. Ouyang, Y . Qiao, and J. Shao, “Octavius: Mitigating task interference in mllms via moe,” in Proc. Int. Conf. Learn. Represent. , 2024

    2024

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 6904–6913

    2017

  6. [14]

    Yin and yang: Balancing and answering binary visual questions,

    P. Zhang, Y . Goyal, D. Summers-Stay, D. Batra, and D. Parikh, “Yin and yang: Balancing and answering binary visual questions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2016, pp. 5014–5022

    2016

  7. [15]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325 , 2015

    2015 arXiv

  8. [16]

    Kosmos-2: Grounding multimodal large language models to the world,

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, and F. Wei, “Kosmos-2: Grounding multimodal large language models to the world,” arXiv preprint arXiv:2306.14824 , 2023

    2023 arXiv

  9. [17]

    BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models,” in Proc. Int. Conf. Mach. Learn. , 2023

    2023

  10. [18]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” arXiv preprint arXiv:2305.06500, 2023

    2023 arXiv

  11. [19]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2024, pp. 26 296–26 306

    2024

  12. [20]

    Protoclip: Prototypical contrastive language image pretraining,

    D. Chen, Z. Wu, F. Liu, Z. Yang, S. Zheng, Y . Tan, and E. Zhou, “Protoclip: Prototypical contrastive language image pretraining,” IEEE Trans. Neural Netw. & Learn. Syst. , 2023

    2023

  13. [21]

    Actionclip: Adapting language-image pretrained models for video action recognition,

    M. Wang, J. Xing, J. Mei, Y . Liu, and Y . Jiang, “Actionclip: Adapting language-image pretrained models for video action recognition,” IEEE Trans. Neural Netw. & Learn. Syst. , 2023

    2023

  14. [22]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following,

    Z. Guo, R. Zhang, X. Zhu, Y . Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li et al. , “Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following,” arXiv preprint arXiv:2309.00615 , 2023

    2023 arXiv

  15. [23]

    Pointllm: Empowering large language models to understand point clouds,

    R. Xu, X. Wang, T. Wang, Y . Chen, J. Pang, and D. Lin, “Pointllm: Empowering large language models to understand point clouds,” inProc. Eur. Conf. Comput. Vis., 2024

    2024

  16. [24]

    3d- llm: Injecting the 3d world into large language models,

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan, “3d- llm: Injecting the 3d world into large language models,” arXiv preprint arXiv:2307.12981, 2023

    2023 arXiv

  17. [25]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark,

    Z. Yin, J. Wang, J. Cao, Z. Shi, D. Liu, M. Li, L. Sheng, L. Bai, X. Huang, Z. Wang et al. , “Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark,” arXiv preprint arXiv:2306.06687, 2023

    2023 arXiv

  18. [26]

    3dmit: 3d multi-modal instruction tuning for scene understanding,

    Z. Li, C. Zhang, X. Wang, R. Ren, Y . Xu, R. Ma, and X. Liu, “3dmit: 3d multi-modal instruction tuning for scene understanding,” arXiv preprint arXiv:2401.03201, 2024

    2024 arXiv

  19. [27]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning,

    S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen, “Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2024, pp. 26 428–26 438

    2024

  20. [28]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers,

    H. Huang, Z. Wang, R. Huang, L. Liu, X. Cheng, Y . Zhao, T. Jin, and Z. Zhao, “Chat-3d v2: Bridging 3d scene and large language models with object identifiers,” arXiv preprint arXiv:2312.08168 , 2023

    2023 arXiv

  21. [29]

    Scenegpt: A language model for 3d scene understanding,

    S. Chandhok, “Scenegpt: A language model for 3d scene understanding,” arXiv preprint arXiv:2408.06926 , 2024

    2024 arXiv

  22. [30]

    Pla: Language- driven open-vocabulary 3d scene understanding,

    R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, “Pla: Language- driven open-vocabulary 3d scene understanding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2023

    2023

  23. [31]

    3d-vista: Pre- trained transformer for 3d vision and text alignment,

    Z. Zhu, X. Ma, Y . Chen, Z. Deng, S. Huang, and Q. Li, “3d-vista: Pre- trained transformer for 3d vision and text alignment,” in Proc. IEEE Int. Conf. Comput. Vis., 2023

    2023

  24. [32]

    Scanqa: 3d question answering for spatial scene understanding,

    D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe, “Scanqa: 3d question answering for spatial scene understanding,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2022

    2022

  25. [33]

    Sqa3d: Situated question answering in 3d scenes,

    X. Ma, S. Yong, Z. Zheng, Q. Li, Y . Liang, S.-C. Zhu, and S. Huang, “Sqa3d: Situated question answering in 3d scenes,” in Proc. Int. Conf. Learn. Represent., 2023

    2023

  26. [34]

    Visrecall: Quantifying information visualisation recallability via question answering,

    Y . Wang, C. Jiao, M. B ˆace, and A. Bulling, “Visrecall: Quantifying information visualisation recallability via question answering,” IEEE Trans. Visual. Comput. Graphics, vol. 28, no. 12, pp. 4995–5005, 2022

    2022

  27. [35]

    Towards explainable 3d grounded visual question answering: A new benchmark and strong baseline,

    L. Zhao, D. Cai, J. Zhang, L. Sheng, D. Xu, R. Zheng, Y . Zhao, L. Wang, and X. Fan, “Towards explainable 3d grounded visual question answering: A new benchmark and strong baseline,” IEEE Trans. Circuits Syst. Video Technol., 2022

    2022

  28. [36]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,

    P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas, “Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes,” in Proc. Eur. Conf. Comput. Vis. , 2020

    2020

  29. [37]

    Scanrefer: 3d object localization in rgb-d scans using natural language,

    D. Z. Chen, A. X. Chang, and M. Niessner, “Scanrefer: 3d object localization in rgb-d scans using natural language,” in Proc. Eur. Conf. Comp. Vis. Springer, 2020, pp. 202–221

    2020

  30. [38]

    Eda: Explicit text-decoupling and dense alignment for 3d visual grounding,

    Y . Wu, X. Cheng, R. Zhang, Z. Cheng, and J. Zhang, “Eda: Explicit text-decoupling and dense alignment for 3d visual grounding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2023

    2023

  31. [39]

    Scan2cap: Context-aware dense captioning in rgb-d scans,

    Z. Chen, A. Gholami, M. Niesner, and A. X. Chang, “Scan2cap: Context-aware dense captioning in rgb-d scans,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2021, pp. 3193–3203

    2021

  32. [40]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niessner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2017

    2017

  33. [41]

    Rio: 3d object instance re-localization in changing indoor environments,

    J. Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Niessner, “Rio: 3d object instance re-localization in changing indoor environments,” in Proc. IEEE Int. Conf. Comput. Vis. , 2019

    2019

  34. [42]

    Matterport3d: Learning from rgb-d data in indoor environments,

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y . Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” arXiv preprint arXiv:1709.06158 , 2017

    2017 arXiv

  35. [43]

    Multimodal learning with transform- ers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transform- ers: A survey,” IEEE Trans. Pattern Anal. Machine Intell. , 2023

    2023

  36. [44]

    A survey on multimodal large language models,

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” arXiv preprint arXiv:2306.13549 , 2023

    2023 arXiv

  37. [45]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention,

    R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y . Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” in Proc. Int. Conf. Learn. Represent. , 2023

    2023

  38. [46]

    Llama-adapter v2: Parameter- efficient visual instruction model,

    P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue, H. Li, and Y . Qiao, “Llama-adapter v2: Parameter- efficient visual instruction model,” arXiv preprint arXiv:2304.15010 , 2023

    2023 arXiv

  39. [47]

    Shikra: Unleashing multimodal llm’s referential dialogue magic,

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,” arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  40. [48]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017

    2017 arXiv

  41. [49]

    Epcl: Frozen clip transformer is an efficient point cloud encoder,

    X. Huang, Z. Huang, S. Li, W. Qu, T. He, Y . Hou, Y . Zuo, and W. Ouyang, “Epcl: Frozen clip transformer is an efficient point cloud encoder,” in Proc. AAAI Conf. Artif. Intell. , 2024

    2024

  42. [50]

    Eva: Exploring the limits of masked visual representation learning at scale,

    Y . Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y . Cao, “Eva: Exploring the limits of masked visual representation learning at scale,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2023, pp. 19 358–19 369

    2023

  43. [51]

    Lavis: A library for language-vision intelligence,

    D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “Lavis: A library for language-vision intelligence,” arXiv preprint arXiv:2209.09019 , 2022

    2022 arXiv

  44. [52]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Int. Conf. Learn. Represent. , 2019. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 14

    2019

  45. [53]

    V otenet: A deep learning label fusion method for multi-atlas segmentation,

    Z. Ding, X. Han, and M. Niethammer, “V otenet: A deep learning label fusion method for multi-atlas segmentation,” in Proc. Int. Conf. Med. Image Comput. Comput.-Assisted Intervention . Springer, 2019, pp. 202–210

    2019

  46. [54]

    Deep modular co-attention networks for visual question answering,

    Z. Yu, J. Yu, Y . Cui, D. Tao, and Q. Tian, “Deep modular co-attention networks for visual question answering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6281–6290

    2019

  47. [55]

    Towards learning a generalist model for embodied navigation,

    D. Zheng, S. Huang, L. Zhao, Y . Zhong, and L. Wang, “Towards learning a generalist model for embodied navigation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2024

    2024

  48. [56]

    Vision-and- dialog navigation,

    J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer, “Vision-and- dialog navigation,” in Conference on Robot Learning . PMLR, 2020, pp. 394–406

    2020

  49. [57]

    Soon: Scenario oriented object navigation with graph-based exploration,

    F. Zhu, X. Liang, Y . Zhu, Q. Yu, X. Chang, and X. Liang, “Soon: Scenario oriented object navigation with graph-based exploration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., June 2021, pp. 12 689– 12 699

    2021

  50. [58]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018

    2018

  51. [59]

    Reverie: Remote embodied visual referring ex- pression in real indoor environments,

    Y . Qi, Q. Wu, P. Anderson, X. Wang, W. Y . Wang, C. Shen, and A. van den Hengel, “Reverie: Remote embodied visual referring ex- pression in real indoor environments,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020

    2020

  52. [60]

    Pointgroup: Dual-set point grouping for 3d instance segmentation,

    L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, “Pointgroup: Dual-set point grouping for 3d instance segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2020

    2020

  53. [61]

    Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,

    J. Yang, X. Chen, S. Qian, N. Madaan, M. Iyengar, D. F. Fouhey, and J. Chai, “Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,” arXiv preprint arXiv:2309.12311 , 2023

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.