Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free VLM agent grounds 3D objects in indoor scenes by pairing 3D and 2D reasoning.

desk verdict Solid, well-engineered zero-shot 3DVG agent with a real architectural advance, but the headline numbers rest on hyperparameters tuned on the same subset they are evaluated on. read the letter →

arxiv 2506.21924 v1 pith:LBNX2EXJ submitted 2025-06-27 cs.CV

Zhao Jin , Rong-Cheng Tu , Jingyi Liao , Wenhao Sun , Xiao Luo , Shunyu Liu , Dacheng Tao This is my paper
classification cs.CV
keywords 3Dvisualgroundingzero-shotvision-languagemodelagentspatialreasoningsemanticverificationpointcloudrendering3D-2Dprojectionretrieval-augmentedfiltering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SPAZER is a zero-shot 3D visual grounding agent that combines 3D spatial reasoning with 2D semantic verification in a progressive pipeline. It renders the point cloud into holistic views, asks a vision-language model to select the most informative view, screens candidate objects through anchor-guided filtering, and then retrieves 2D camera frames for the remaining candidates to make a joint 3D-2D decision. The paper reports that this training-free pipeline outperforms previous zero-shot methods by about 9% on ScanRefer and 10.9% on Nr3D, and that on Nr3D it comes within roughly one point of fully supervised baselines. The authors argue that existing zero-shot methods specialize in either spatial or semantic understanding, and that grounding requires both.

What carries the argument

The load-bearing mechanism is a coarse-to-fine cascade of four modules. First, holistic multi-view rendering places a bird's-eye view and n oblique views around the scene and lets the VLM pick the view where the query object is most visible. Second, retrieval-augmented anchor filtering matches the queried object class to detector categories by text similarity and, when confidence is low, builds a visual object table from cropped object images so the VLM can filter candidates visually. Third, Top-k candidate screening annotates the selected 3D view with object IDs and asks the VLM for the k most plausible matches. Fourth, 3D-2D joint decision-making projects each candidate's 3D bounding box into the scanned camera images, maps the box corners through camera intrinsics and extrinsics, selects the most informative view per candidate, and prompts the VLM to decide using both the global 3D rendering and the 2D close-ups with consistent object IDs.

What would settle it

Run SPAZER on a ScanRefer or Nr3D subset with the off-the-shelf detector's boxes and class labels replaced by ground-truth boxes and class labels; the accuracy gap between the two runs quantifies how much of the claimed performance is carried by the detector rather than by the spatial-semantic reasoning pipeline. If the gap is large, the central 'zero-shot reasoning' claim is mostly a detector artifact; if it is small, the reasoning pipeline is the dominant factor.

Watch

Extended reading notes

Core claim

The central claim is that the gap between zero-shot and supervised 3D visual grounding can be closed, without any training on 3D-labeled data, by decoupling spatial and semantic reasoning into separate stages that use different modalities. SPAZER first reasons about where the target could be using rendered views of the 3D scene itself, rather than textual coordinates derived from predicted boxes, and then verifies appearance-level attributes such as color and material using 2D camera images of the candidate objects. The authors report state-of-the-art zero-shot accuracy on ScanRefer and Nr3D, with the 2D verification stage contributing the largest single accuracy gain (an average of 11.6 points over 3D-only decision-making) in their ablations.

Load-bearing premise

SPAZER's candidate pool is supplied entirely by an off-the-shelf 3D instance segmenter (Mask3D, pretrained on ScanNet), so if the target object is missed or misclassified by this detector it can never enter the candidate list and the whole pipeline fails regardless of VLM reasoning.

Editorial extensions

If this is right

  • On ScanRefer, SPAZER reaches 57.2% Acc@0.25 overall, exceeding all prior zero-shot methods and matching or beating several fully supervised baselines such as BUTD-DETR and 3DVG-Transformer.
  • On Nr3D, SPAZER achieves 63.8% overall accuracy without ground-truth class labels, and 73.2% with them, which the authors describe as within about one point of supervised state of the art.
  • Removing the 2D verification stage drops accuracy by an average of 11.6 points, indicating that 3D spatial reasoning alone is not enough and that the two modalities are complementary.
  • Replacing the default VLM with a stronger vision-language model raises overall accuracy on Nr3D from 63.8% to 64.3%, suggesting the framework benefits directly from VLM progress.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the detector ceiling is the binding constraint, then pairing SPAZER with a stronger open-vocabulary 3D segmenter should produce larger gains than swapping the VLM, which would make instance segmentation the primary lever for further improvement.
  • The same spatial-first, semantic-verify-later cascade could be adapted to live embodied settings where the 2D camera stream is generated in real time rather than taken from a pre-scanned video, a testable extension the paper does not explore.
  • Because the paper's headline gains are measured on 250-sample subsets of ScanRefer and Nr3D, a full-dataset evaluation with the default VLM would be needed to confirm that the reported margins hold exactly as stated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SPAZER is a zero-shot 3D visual grounding agent that combines 3D holistic rendered views with 2D camera images in a progressive spatial-semantic reasoning pipeline. The method first selects an optimal 3D viewpoint, screens candidate objects via anchor filtering and Top-k selection, and then performs joint 3D-2D decision-making using GPT-4o as the core VLM. The paper reports large gains over prior zero-shot methods on ScanRefer and Nr3D and claims near-supervised accuracy on Nr3D.

Significance. The proposed pipeline is well-motivated and clearly described; the integration of global 3D context with 2D close-up verification is a sensible complement to existing 3D- and 2D-only paradigms. The paper provides complete prompt templates in the appendix, which aids reproducibility, and it includes an explicit error-type analysis. If substantiated on matched full-set evaluations, the results would make SPAZER a strong new state of the art for zero-shot 3DVG. However, the current evaluation protocol does not fully support the headline quantitative claims.

major comments (4)
  1. [Sec. 4.4 / Fig. 4 and Eq. (3)] The hyperparameters k, n, and the threshold τ are selected using the same 250-sample Nr3D subset on which the final results in Table 2 are reported. Figure 4 chooses k=4 and n=4 based on this subset, and Eq. (3) introduces τ without a separate validation split. This selection-on-test protocol can inflate the reported gains over prior zero-shot methods and the 63.8% overall accuracy. Please split the subset into validation and test portions (or use cross-validation) for hyperparameter selection and report final accuracy on a held-out test portion.
  2. [Sec. 4.1 / Table 2] The comparison with supervised methods is not matched in evaluation set. Supervised numbers (e.g., MiKASA, SceneVerse, 3D-VisTA) are from the full Nr3D benchmark, while SPAZER is evaluated on the 250-sample subset used by VLM-Grounder. The Introduction's claim of a 'marginal gap (~1%)' to supervised SOTA is therefore not a like-for-like comparison; the gap could be substantially larger on the full benchmark. Please either run SPAZER on the full Nr3D and ScanRefer sets or report supervised baselines on the same subset before making this claim.
  3. [Appendix B.2 / Table 9] The full-dataset consistency check is performed only with Qwen2-VL-72B and Qwen2.5-VL-72B, not with the default GPT-4o that produces the headline results (Sec. 4.2, Table 4). The observed <2.0-point full-versus-subset differences for Qwen models do not establish the same behavior for GPT-4o. Please provide at least one full-set evaluation with GPT-4o, or justify why the Qwen results are sufficient to rule out subset-specific effects.
  4. [Appendix A] All reported numbers are from single runs at temperature 0.2. Since VLM outputs are stochastic, the claimed margins (e.g., +15.8 overall on Nr3D over VLM-Grounder in Table 2, and +9.0 on ScanRefer Acc@0.5 over CSVG in Table 1) may lie within run-to-run noise. Please report the mean and standard deviation over at least three runs, or otherwise bound the variance, to support the quantitative claims.
minor comments (5)
  1. [Abstract / Introduction] The '9% on ScanRefer' gain is not tied to a specific metric; Table 1 shows the gain over CSVG in Overall Acc@0.5 is 9.0 points, while Acc@0.25 gains are different. Please specify the metric in the abstract.
  2. [Tables 1 and 2] Add a note that all zero-shot methods are evaluated on the 250-sample subset, to clarify the comparison basis with supervised methods.
  3. [Appendix B.2 / Fig. 5] The appendix contains corrupted placeholder glyphs (e.g., lines beginning '/uni00000017/...' before Fig. 5 and in Fig. 6). These should be rendered properly before submission.
  4. [Fig. 4] The label text in panel (a) appears garbled in the current PDF; please ensure the figure is readable.
  5. [Sec. 3.1] The sentence 'Our proposed agent SPAZER solve this task' should read 'SPAZER solves this task.'

Circularity Check

1 steps flagged · score 4.0 of 10

Main Nr3D/ScanRefer accuracies are reported on the same subset used to tune k, n, and tau; otherwise the pipeline is not circular and self-citations are not load-bearing.

  1. fitted input called prediction [Sec. 4.1 (Experimental setup), Sec. 4.4 (Ablation study, Fig. 4), and Eq. (3)]
    "To enable fair comparison and reduce expenditure, our main experiments are conducted on the same ScanRefer and Nr3D subsets as [41]. ... All ablation studies are conducted on the same subset of Nr3D as [41]. ... The number of views n is set to 4, and the Top-k parameter is set to k = 4."

    The headline accuracies in Tables 1 and 2 (including the 9.0%/10.9% gains and the near-supervised Nr3D claim) are computed on the same 250-sample subsets on which the method's free parameters are chosen. Section 4.4 selects k=4 and n=4 by inspecting overall accuracy on 'the same subset of Nr3D as [41]', and the threshold tau in Eq. (3) is introduced without a specified held-out validation split. Thus the reported numbers are in-sample with respect to these hyperparameter choices rather than independent predictions; the gains are partly an artifact of selecting the configuration that maximizes accuracy on the evaluation set.

full rationale

SPAZER is an empirical agent pipeline with no mathematical derivation, so the relevant circularity question is whether the reported evaluation reduces to the inputs used to build it. The only concrete instance is the selection of k, n, and tau by ablation on the same 250-sample Nr3D subset used for the final accuracy and SOTA comparisons; this is a fitted-on-test evaluation issue, not a definitional equivalence. No load-bearing self-citation was found: the authors' prior works [32,33,34] appear only as related-work context, and the neuroscience framing cites external work. The Mask3D dependence is acknowledged in the Limitations and Appendix C.1 as a detection ceiling and applies to competing detect-and-match baselines as well. The paper also reports full-vs-subset checks showing under-2.0-point variations for open VLMs, which mitigates but does not eliminate the in-sample concern for GPT-4o. Overall, the core pipeline content (holistic view selection, anchor filtering, 3D-2D joint decision) is not forced by self-citation or by construction; the score reflects the partial circularity of tuning and evaluating on the same benchmark subset.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of pretrained perception tools (Mask3D), the perceptual and reasoning capability of a proprietary VLM (GPT-4o), and dataset-specific assumptions (availability of registered 2D images and calibration). No new physical entities or mathematical axioms are introduced. The hyperparameters k, n, and tau are tuned on the evaluation subset, which is the main self-referential element.

free parameters (3)
  • Top-k candidate count (k) = 4
    Selected via ablation on the Nr3D evaluation subset (Fig. 4a); affects candidate coverage and final decision accuracy.
  • Number of additional rendered views (n) = 4
    Selected via ablation on the Nr3D evaluation subset (Fig. 4b); more views help until redundancy hurts.
  • Anchor filtering threshold (tau) = not reported
    Threshold in Eq. 3 for choosing text-based vs vision-based class filtering; set by hand, exact value not disclosed.
assumptions (5)
  • domain assumption Mask3D provides reliable 3D instance segmentation on ScanNet scenes.
    Used in Sec. 4.1 to obtain object bounding boxes and class labels; errors propagate to grounding results.
  • domain assumption 2D camera images and camera intrinsics/extrinsics are available and registered with the 3D scan.
    Section 3.4 relies on this for projecting 3D boxes to 2D views; true for ScanNet but not for all 3D datasets.
  • domain assumption The VLM (GPT-4o) can perform view selection, candidate screening, and joint reasoning from rendered and camera images.
    The entire agent is built on this capability (Secs. 3.2 to 3.4); no training of the VLM is performed.
  • domain assumption The 250-sample evaluation subset is representative of the full benchmark.
    Main results use the same subset as VLM-Grounder; Appendix B.2 checks consistency with the full set for open VLMs, but not for GPT-4o.
  • domain assumption Point cloud rendering produces images that preserve enough spatial and object detail for VLM interpretation.
    Section 3.2 introduces holistic rendering as the 3D representation; if rendering loses fidelity, spatial reasoning degrades.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding." pith.science (2026). https://pith.science/paper/LBNX2EXJ

@misc{pith2026250621924,
  author       = {Pith},
  title        = {Pith review of: SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBNX2EXJ}},
  note         = {Machine review of arXiv:2506.21924}
}
read the original abstract

3D Visual Grounding (3DVG) aims to localize target objects within a 3D scene based on natural language queries. To alleviate the reliance on costly 3D training data, recent studies have explored zero-shot 3DVG by leveraging the extensive knowledge and powerful reasoning capabilities of pre-trained LLMs and VLMs. However, existing paradigms tend to emphasize either spatial (3D-based) or semantic (2D-based) understanding, limiting their effectiveness in complex real-world applications. In this work, we introduce SPAZER - a VLM-driven agent that combines both modalities in a progressive reasoning framework. It first holistically analyzes the scene and produces a 3D rendering from the optimal viewpoint. Based on this, anchor-guided candidate screening is conducted to perform a coarse-level localization of potential objects. Furthermore, leveraging retrieved relevant 2D camera images, 3D-2D joint decision-making is efficiently performed to determine the best-matching object. By bridging spatial and semantic reasoning neural streams, SPAZER achieves robust zero-shot grounding without training on 3D-labeled data. Extensive experiments on ScanRefer and Nr3D benchmarks demonstrate that SPAZER significantly outperforms previous state-of-the-art zero-shot methods, achieving notable gains of 9.0% and 10.9% in accuracy.

Figures

Figures reproduced from arXiv: 2506.21924 by the authors.

Figure 1
Figure 1. Schematic comparison between our proposed SPAZER and existing zero-shot 3DVG [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SPAZER framework. Given input 3D scene and a query text of the target [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of 3DVG results. Incorrectly predicted 3D bounding boxes are [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation studies on parameter k and n. Effect of k in Top-k. In the can￾didate screening stage, our agent gives the Top-k most possible tar￾get object IDs. We evaluate the Top-k accuracy (i.e., whether the ground-truth object is in the k can￾didates) and the overall gr…
Figure 5
Figure 5. Figure 5: Error type distribution on ScanRefer dataset. Ours (3D only) indicates that our agent selects [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Typical types of failure cases. The prediction and ground-truth are highlighted in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TDVR: Joint Text Disambiguation and Viewpoint Reasoning for Zero-Shot 3D Visual Grounding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A training-free pipeline that disambiguates text queries and infers viewpoints improves zero-shot 3D visual grounding, reaching 64.06% Acc@0.5 on ScanRefer.

Reference graph

Works this paper leans on

54 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    ReferIt3D: Neural listeners for fine-grained 3D object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. ReferIt3D: Neural listeners for fine-grained 3D object identification in real-world scenes. In European Conference on Computer Vision (ECCV), pages 422–440, 2020

    work page 2020

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3DJCG: A unified framework for joint dense captioning and visual grounding on 3D point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16464–16473, 2022

    work page 2022

  4. [4]

    Spatial and object visualization cognitive styles: Validation studies in 3800 individuals

    Christopher F Chabris, Thomas E Jerde, Anita W Woolley, Margaret E Gerbasi, Jonathon P Schuldt, Sean L Bennett, J Richard Hackman, and Stephen M Kosslyn. Spatial and object visualization cognitive styles: Validation studies in 3800 individuals. Group brain technical report, 2:2, 2006

    work page 2006

  5. [5]

    MiKASA: Multi-key-anchor & scene-aware transformer for 3D visual grounding

    Chun-Peng Chang, Shaoxiang Wang, Alain Pagani, and Didier Stricker. MiKASA: Multi-key-anchor & scene-aware transformer for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14131–14140, 2024

    work page 2024

  6. [6]

    ScanRefer: 3D object localization in RGB-D scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. ScanRefer: 3D object localization in RGB-D scans using natural language. In European Conference on Computer Vision (ECCV), pages 202–221, 2020

    work page 2020

  7. [7]

    D3Net: A unified speaker-listener architecture for 3D dense captioning and visual grounding

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X Chang. D3Net: A unified speaker-listener architecture for 3D dense captioning and visual grounding. In European Conference on Computer Vision (ECCV), pages 487–505. Springer, 2022

    work page 2022

  8. [8]

    Language conditioned spatial relation reasoning for 3D object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3D object grounding. Advances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

Show all 54 references
  1. [9]

    ScanNet: Richly-annotated 3D reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5828–5839, 2017

    2017

  2. [10]

    Transcrib3D: 3D referring expression resolution through large language models

    Jiading Fang, Xiangshan Tan, Shengjie Lin, Igor Vasiljevic, Vitor Guizilini, Hongyuan Mei, Rares Ambrus, Gregory Shakhnarovich, and Matthew R Walter. Transcrib3D: 3D referring expression resolution through large language models. In IEEE/RSJ International Conference on Intellig...

    2024

  3. [11]

    Text-guided sparse voxel pruning for efficient 3D visual grounding

    Wenxuan Guo, Xiuwei Xu, Ziwei Wang, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Text-guided sparse voxel pruning for efficient 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  4. [12]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2023

    2023

  5. [13]

    Chat-Scene: Bridging 3D scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-Scene: Bridging 3D scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Pr...

    2024

  6. [14]

    Multi-view transformer for 3D visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15524–15533, 2022

    2022

  7. [15]

    Bottom up top down detection transformers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision (ECCV), pages 417–433. Springer, 2022

    2022

  8. [16]

    SceneVerse: Scaling 3D vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. SceneVerse: Scaling 3D vision-language learning for grounded scene understanding. In European Conference on Computer Vision (ECCV), pages 289–310, 2024

    2024

  9. [17]

    PointGroup: Dual-set point grouping for 3D instance segmentation

    Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. PointGroup: Dual-set point grouping for 3D instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4867–4876, 2020. 10

    2020

  10. [18]

    Context-aware alignment and mutual masking for 3D-language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3D-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10984–10994, 2023

    2023

  11. [19]

    SeeGround: See and ground for zero-shot open-vocabulary 3D visual grounding

    Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. SeeGround: See and ground for zero-shot open-vocabulary 3D visual grounding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  12. [20]

    A survey on text-guided 3D visual grounding: elements, recent advances, and future directions

    Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. A survey on text-guided 3D visual grounding: elements, recent advances, and future directions. arXiv preprint arXiv:2406.05785, 2024

    2024 arXiv

  13. [21]

    ScanERU: Interactive 3D visual grounding based on embodied reference understanding

    Ziyang Lu, Yunqiang Pei, Guoqing Wang, Peiwei Li, Yang Yang, Yinjie Lei, and Heng Tao Shen. ScanERU: Interactive 3D visual grounding based on embodied reference understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3936–3944, 2024

    2024

  14. [22]

    3D-SPS: Single-stage 3D visual grounding via referred point progressive selection

    Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3D-SPS: Single-stage 3D visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1...

    2022

  15. [23]

    Large language model agent: A survey on methodology, applications and challenges

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460, 2025

    2025 arXiv

  16. [24]

    Evolving symbolic 3D visual grounder with weakly supervised reflection

    Boyu Mi, Hanqing Wang, Tai Wang, Yilun Chen, and Jiangmiao Pang. Evolving symbolic 3D visual grounder with weakly supervised reflection. arXiv preprint arXiv:2502.01401, 2025

    2025 arXiv

  17. [25]

    OpenAI. ChatGPT. https://chat.openai.com/

  18. [26]

    Proxytransformation: Preshaping point cloud manifold with proxy attention for 3D visual grounding

    Qihang Peng, Henry Zheng, and Gao Huang. Proxytransformation: Preshaping point cloud manifold with proxy attention for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  19. [27]

    OpenScene: 3D scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. OpenScene: 3D scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–824, 2023

    2023

  20. [28]

    Deep hough voting for 3D object detection in point clouds

    Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9277–9286, 2019

    2019

  21. [29]

    GPT4Scene: Understand 3D scenes from videos with vision-language models

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. GPT4Scene: Understand 3D scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025

    2025 arXiv

  22. [30]

    Mask3D: Mask transformer for 3D semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask transformer for 3D semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023

    2023

  23. [31]

    HuggingGPT: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems (NeurIPS), 36:38154–38180, 2023

    2023

  24. [32]

    SPAgent: Adaptive task decomposition and model selection for general video generation and editing

    Rong-Cheng Tu, Wenhao Sun, Zhao Jin, Jingyi Liao, Jiaxing Huang, and Dacheng Tao. SPAgent: Adaptive task decomposition and model selection for general video generation and editing. arXiv preprint arXiv:2411.18983, 2024

    2024 arXiv

  25. [33]

    Mllm- guided vlm fine-tuning with joint inference for zero-shot composed image retrieval

    Rong-Cheng Tu, Zhao Jin, Jingyi Liao, Xiao Luo, Yingjie Wang, Li Shen, and Dacheng Tao. Mllm- guided vlm fine-tuning with joint inference for zero-shot composed image retrieval. arXiv preprint arXiv:2505.19707, 2025

    2025 arXiv

  26. [34]

    Multimodal reasoning agent for zero-shot composed image retrieval

    Rong-Cheng Tu, Wenhao Sun, Hanzhe You, Yingjie Wang, Jiaxing Huang, Li Shen, and Dacheng Tao. Multimodal reasoning agent for zero-shot composed image retrieval. arXiv preprint arXiv:2505.19952, 2025

    2025 arXiv

  27. [35]

    ROSS3D: Reconstructive visual instruction tuning with 3D-awareness

    Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. ROSS3D: Reconstructive visual instruction tuning with 3D-awareness. arXiv preprint arXiv:2504.01901, 2025

    2025 arXiv

  28. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 11

    2024 arXiv

  29. [37]

    EmbodiedScan: A holistic multi-modal 3D perception suite towards embodied AI

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. EmbodiedScan: A holistic multi-modal 3D perception suite towards embodied AI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  30. [38]

    LIBA: Language instructed multi-granularity bridge assistant for 3D visual grounding

    Yuan Wang, Ya-Li Li, WU Eastman ZY , and Shengjin Wang. LIBA: Language instructed multi-granularity bridge assistant for 3D visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8114–8122, 2025

    2025

  31. [39]

    Visual ChatGPT: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023

    2023 arXiv

  32. [40]

    EDA: Explicit text-decoupling and dense alignment for 3D visual grounding

    Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. EDA: Explicit text-decoupling and dense alignment for 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19231–19242, 2023

    2023

  33. [41]

    VLM-Grounder: A VLM agent for zero-shot 3D visual grounding

    Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. VLM-Grounder: A VLM agent for zero-shot 3D visual grounding. In Conference on Robot Learning (CoRL), 2024

    2024

  34. [42]

    LLM-Grounder: Open-vocabulary 3D visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. LLM-Grounder: Open-vocabulary 3D visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 76...

    2024

  35. [43]

    Thinking in Space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  36. [44]

    Inst3D-LMM: Instance-aware 3D scene understanding with multi-modal instruction tuning

    Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3D-LMM: Instance-aware 3D scene understanding with multi-modal instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  37. [45]

    Solving zero-shot 3D visual grounding as constraint satisfaction problems

    Qihao Yuan, Jiaming Zhang, Kailai Li, and Rainer Stiefelhagen. Solving zero-shot 3D visual grounding as constraint satisfaction problems. arXiv preprint arXiv:2411.14594, 2024

    2024 arXiv

  38. [46]

    In- stanceRefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stanceRefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Con...

    2021

  39. [47]

    Visual programming for zero-shot open-vocabulary 3D visual grounding

    Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open-vocabulary 3D visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20623–20633, 2024

    2024

  40. [48]

    Towards CLIP-driven language-free 3D visual grounding via 2D-3D relational enhancement and consistency

    Yuqi Zhang, Han Luo, and Yinjie Lei. Towards CLIP-driven language-free 3D visual grounding via 2D-3D relational enhancement and consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13063–13072, 2024

    2024

  41. [49]

    3DVG-Transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3DVG-Transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2928–2937, 2021

    2021

  42. [50]

    Video-3D LLM: Learning position-aware video representation for 3D scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3D LLM: Learning position-aware video representation for 3D scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  43. [51]

    unknown" for the category. Your response should be formatted in JSON. Here are some examples: Input: Query: this is a brown cabinet. it is to the right of a picture. Output: {

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3D-VISTA: Pre-trained transformer for 3D vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2911–2921, 2023. 12 SPAZER: Spatial-Semantic...

    2023

  44. [52]

    Start with the global view image: • Analyze the overall spatial layout and object distribution in the room. • Use the global view to evaluate view-independent spatial relationships, which do not rely on a specific viewpoint: Examples include: near, close to, next to, far, abov...

  45. [53]

    • Evaluate visual features: color, shape, size, texture, and material

    Then examine the camera images: • Validate candidate objects identified from the global view. • Evaluate visual features: color, shape, size, texture, and material. • Use camera views to judge view-dependent spatial relationships, which depend on the camera perspective: Exampl...

  46. [54]

    {text}" Object IDs to choose from: {object_id_list} Output format (JSON): {

    Iterate if needed: • If no candidate fully matches the query, return to the global view and reassess alterna- tives. • Repeat verification with camera images until you confidently identify the best match. Task: Select the object ID of the target class: {target_class} Query des...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.