Pith. sign in

REVIEW 5 major objections 3 minor 11 cited by

3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

T0 review · 5 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a cold-start CoT dataset plus GRPO reinforcement learning with perception, semantic, and format rewards lets a single 3D VLM beat prior state-of-the-art across seven scene-understanding tasks by an average of 10%.

desk verdict Strong 3D VLM systems paper with a load-bearing contamination risk: Scene-30K may overlap the evaluation benchmarks, and view weights are selected on test. read the letter →

arxiv 2507.23478 v1 pith:ERF45AGU submitted 2025-07-31 cs.CV

Ting Huang , Zeyu Zhang , Hao Tang This is my paper · ORCID
classification cs.CV
keywords 3Dvision-languagemodelchain-of-thoughtreinforcementlearningGRPOdensecaptioningvisualgroundingquestionansweringdynamicviewselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a 3D vision-language model can be turned into a strong generalist reasoner across seven scene-understanding tasks by combining a cold-start supervised fine-tuning stage on a new 30K chain-of-thought dataset with a GRPO reinforcement-learning stage that rewards spatial perception, semantic fidelity, and output formatting. The authors report an average 10% improvement over prior state-of-the-art on benchmarks covering dense captioning, object captioning, question answering, dialogue, visual grounding, situated reasoning, and planning. The interest of the claim is that a single architecture, built on a frozen 7B VLM with about 142M trainable parameters, can cover perception-heavy tasks and open-ended reasoning tasks without task-specific heads. If the claim holds, it supports a recipe of cold-start CoT plus reward-shaped RL plus adaptive view selection as a general way to improve reasoning in 3D VLMs.

What carries the argument

The engine of the method is the combination of the Scene-30K cold start and three GRPO rewards. Scene-30K provides supervised chain-of-thought demonstrations in the fixed think/answer format, giving the base model a stable starting policy. During reinforcement learning, the format reward enforces output structure, the perception reward (IoU between predicted and ground-truth boxes) keeps the model spatially grounded, and the semantic similarity reward (CLIP cosine similarity between predicted and reference answers) keeps answers semantically faithful. Dynamic view selection adds a learned ranking over candidate views, fusing text-relevance, coverage, and CLIP scores with learnable weights, so the model sees the most informative 2D renderings of the 3D scene.

What would settle it

Compare the question texts and scene IDs in Scene-30K against the validation and test splits of ScanQA, ScanRefer, Nr3D, and SceneVerse. If any validation or test question appears in Scene-30K, the reported improvements could be explained by train/test leakage rather than by the method. A second check is to retrain 3D-R1 with Scene-30K built exclusively from training splits and confirm that the average 10% gain persists.

Watch

Extended reading notes

Core claim

3D-R1 is a unified 3D vision-language model that answers 3D scene queries in a structured <think>...</think><answer>...</answer> format. The central discovery is that pure reinforcement learning is unstable on a 3D VLM, so the authors first fine-tune on Scene-30K, a machine-generated dataset of 30K question-chain-of-thought-answer triples produced by feeding scene descriptions to Gemini 2.5 Pro, then apply GRPO with three rewards: a format reward for structured output, a perception reward measuring bounding-box intersection-over-union, and a semantic-similarity reward based on CLIP text-embedding cosine similarity. A third component selects informative 2D views of the point cloud by learning to fuse text-to-3D, image-to-3D, and CLIP alignment scores. The paper reports state-of-the-art results on ScanRefer, Nr3D, ScanQA, Cap3D, SQA3D, and the 3D-LLM dialogue and planning benchmarks, with an average improvement of about 10% over prior methods.

Load-bearing premise

The reported gains depend on Scene-30K being built only from training splits of the benchmark datasets; the paper never states this, so if any validation or test questions leaked into the synthetic training data, the state-of-the-art scores would be invalid.

Editorial extensions

If this is right

  • A single 3D-R1 model, without task-specific heads, can handle dense captioning, object captioning, QA, dialogue, grounding, reasoning, and planning, with state-of-the-art results on each of the tested benchmarks.
  • The cold-start CoT plus GRPO schema carries the DeepSeek-R1-style reinforcement-learning reasoning boost from text-only LLMs into 3D vision-language models, where pure RL alone is reported to be unstable.
  • The three-reward design ties reasoning improvement to spatial grounding: the perception reward connects the model's generated text to bounding-box predictions, so better reasoning comes together with better localization.
  • Dynamic view selection beats fixed view baselines such as all views, horizontal views, or bottom views, showing that learning which perspectives to attend is a measurable part of the performance gain.
  • The recipe is parameter-efficient: about 142M trainable parameters on a frozen roughly-7B base VLM produce the reported gains across all tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same cold-start CoT plus GRPO recipe could likely transfer to other 3D representations such as Gaussian splats or meshes, and to other base VLMs, potentially reproducing part of the average 10% gain on new tasks.
  • The view-selection module, trained on static scenes with a fixed view pool, could be extended to active camera control for embodied agents, letting the model decide where to look next rather than ranking a fixed set of renderings.
  • Because the perception reward uses IoU, the method's success suggests that grounding reinforcement-learning rewards in explicit spatial signals is a practical defense against reward hacking in vision-language RL, a design principle worth testing in other multimodal RL pipelines.
  • If Scene-30K is released and expanded, it would become a reusable cold-start CoT dataset for 3D reasoning, lowering the barrier for future 3D-VLM reasoning research beyond the benchmarks used here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper introduces 3D-R1, a 3D vision-language model that combines a synthetic chain-of-thought cold-start dataset (Scene-30K), GRPO-based reinforcement learning with perception, semantic similarity, and format rewards, and a learned dynamic view selection module. The authors report state-of-the-art results across seven 3D scene understanding tasks, including dense captioning, object captioning, question answering, visual grounding, dialogue, reasoning, and planning, with an claimed average improvement of 10%. The paper includes ablations for reward components, view selection, modality encoders, and LoRA rank, and releases code and a project website.

Significance. If the reported results are trustworthy, the paper makes a valuable contribution by demonstrating a general recipe for improving 3D VLMs: synthetic CoT cold-start followed by RL with task-relevant rewards, plus adaptive view selection. The breadth of tasks covered, the relatively lightweight training (only ~142M trainable parameters), and the public release of code and models are concrete strengths. The ablation study on reward components is particularly useful for the community. However, several load-bearing technical details and potential evaluation issues need to be addressed before the central claims can be accepted.

major comments (5)
  1. [Datasets and CoT Data Engine (Table 1) and Experiment (Table 3)] The Scene-30K dataset is explicitly constructed from ScanQA, ScanRefer, Nr3D, and SceneVerse, and the evaluation then uses ScanQA validation and test sets, ScanRefer, and Nr3D. The paper never states that only training splits of these source datasets were used to synthesize Scene-30K. If any validation/test questions or scenes leaked into Scene-30K, the reported state-of-the-art numbers (e.g., Table 3) would reflect memorization rather than generalization. The paper must provide split-level provenance for Scene-30K and explicitly rule out overlap with the evaluation sets.
  2. [Reinforcement Learning, Eq. (2)] The perception reward Rp = IoU(b, b*) requires a predicted bounding box b*, but the model is described only as an autoregressive text decoder that outputs text answers. No output head, parsing rule, or other mechanism for producing b* is specified anywhere. This is load-bearing: Table 7 shows the perception reward contributes substantially to the final gains, yet the algorithm that computes it is not defined. The authors must specify how b* is obtained and how the reward is computed for every training sample.
  3. [Ablation Study, Table 9 and Dynamic View Selection, Eq. (7)] The paper claims the view fusion weights w_t, w_c, w_clip are learned dynamically (Section "Dynamic View Selection", Eq. (7)), but Table 9 presents a grid search over fixed weight values evaluated directly on ScanQA and ScanRefer. If those are the same evaluation sets used for the main results, then the final numbers are selected based on test-set performance, which is circular. The authors should clarify whether Table 9 was obtained on a held-out validation split, or if the main results use weights fixed before evaluation.
  4. [Main Results, Table 4] Table 4 reports human A/B test results in which 3D-R1 obtains Win% = 34.56 and Lose% = 65.34 for caption quality, and Win% = 27.34 and Lose% = 69.12 for hallucination. These numbers indicate that human evaluators strongly prefer the comparison method over 3D-R1, which directly contradicts the text's claim that 3D-R1 'achieves the highest scores across all evaluation criteria'. The table or its interpretation must be corrected.
  5. [Abstract and Datasets and Metrics] The abstract claims an 'average improvement of 10% across various 3D scene benchmarks', but no precise definition of the average is given. Improvements in individual tables range from a few percent (Table 5, Dialogue CIDEr 280.34 vs 270.10) to over 10 points (Table 2, ScanRefer C@0.25 91.85 vs 85.42). The aggregation method, the set of baselines used for the average, and how incomparable metrics are combined must be specified.
minor comments (3)
  1. [CoT Data Engine (Figure 3)] The scene description generator is described only as 'a pretrained 3D VLM' without naming the model or providing details on how it was trained. This is an important reproducibility detail for constructing Scene-30K.
  2. [Table 4 and Table 8] The 'Bottom 6-views' row reports a CLIP R@1 of 4.60 in Table 4 and 6.63 in Table 8, which is implausibly low compared to the horizontal and all-views settings. This suggests a possible unit error or a mistaken metric, and the authors should check these numbers.
  3. [Throughout] The text contains several inconsistent renderings of method names, e.g., 'V ote2Cap-DETR' and 'LLaV A-3D', and the paper would benefit from a careful proofreading pass.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial circularity: view-fusion weights are grid-searched on the evaluation benchmarks and then reported as 'learned', making the view-selection gains partly fit to the test signal; Scene-30K split provenance is also never stated.

  1. fitted input called prediction [Dynamic View Selection (Eq. 7) and Ablation Study, Table 9]
    "Instead of manually tuning their relative importance, we dynamically learn the weight of these components. We introduce learnable parameters w_t, w_c, w_clip for the text relevance, coverage, and CLIP alignment scores, respectively. ... Table 9 presents a grid search for various weight combinations. The results show that all three cues are complementary: using any single score alone yields suboptimal results, while balanced weighting (wt = 0.3,wc = 0.5,wclip = 0.5) achieves the best performance across tasks."

    The paper presents the view-fusion weights as learned parameters (Eq. 7) and then reports the best configuration from a grid search evaluated directly on ScanQA and ScanRefer, the same benchmarks used for the SOTA claims. The 'Learned 6-view selection' numbers in Table 8 are therefore maxima over a grid fitted to the evaluation signal, not independent predictions of a learned view-selection module. The claimed improvement from dynamic view selection is partly forced by construction because the weights were selected on the exact metrics and datasets used to report the gain.

full rationale

The core training pipeline—Scene-30K cold-start SFT, GRPO with format/perception/semantic rewards, and the encoder/LoRA architecture—is not circular by construction; the reward functions and evaluation metrics overlap (CLIP similarity, IoU), but optimizing a surrogate of the metric on training data is standard practice. The one concrete circular step is the dynamic view-fusion weights: they are introduced as learnable but selected by grid search on ScanQA and ScanRefer, making the view-selection contribution a fit to the evaluation benchmarks rather than an independent inference. A second, load-bearing gap is missing support: the paper says Scene-30K is synthesized from ScanQA, ScanRefer, Nr3D, and SceneVerse and then evaluates on those same benchmarks, but never states that only training splits were used; if any evaluation question or scene entered Scene-30K, the reported SOTA would be memorization. This is a contamination risk rather than a demonstrated equivalence, so it is not counted as a separate circular step, but it materially weakens the claim that the improvements are generalization. Self-citations to prior 3D CoCa / DC-Scene work are related-work mentions and not load-bearing. Overall, partial circularity in a sub-claim with independent content elsewhere, so score 5.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces a new dataset (Scene-30K) and a trained model, but no new physical entities, forces, or conserved quantities. The free parameters and axioms listed above are the main hidden inputs on which the central claim depends.

free parameters (4)
  • View fusion weights w_t, w_c, w_clip = w_t=0.3, w_c=0.5, w_clip=0.5 (grid search result)
    Selected by grid search over evaluation metrics in Table 9, not learned during training as claimed. This hand-tuned constant directly influences the reported improvements.
  • LoRA rank delta = 12
    Chosen by ablation in Table 11; higher ranks give diminishing returns. It is a capacity hyperparameter, but it is selected on the evaluation sets rather than by a principled criterion.
  • Filtering thresholds in Algorithm 1 = min think words 30, min answer words 20, min steps 3, Levenshtein threshold 0.8
    Ad hoc thresholds that determine the composition of Scene-30K and thus the cold-start model; chosen by hand without a principled basis or sensitivity analysis.
  • L2 target mu for w_t = 0.3
    Target value for the L2 regularization on the text relevance weight in dynamic view selection; hand-chosen and then validated by grid search (Table 9).
assumptions (5)
  • domain assumption CoT traces generated by Gemini 2.5 Pro from a scene text description are valid, factually grounded reasoning supervision for 3D VLMs.
    The entire cold-start stage relies on this. The rule-based filter checks format, length, step count, and self-consistency but never verifies factual correctness against the 3D scene.
  • domain assumption The benchmark evaluation is not contaminated: Scene-30K is created only from training splits of ScanQA/ScanRefer/Nr3D/SceneVerse and not from validation/test splits.
    The paper uses the same benchmark names for data synthesis and evaluation and never states the split separation.
  • ad hoc to paper The model's text output can be parsed into a predicted bounding box, enabling the IoU perception reward in Eq. (2).
    No parsing mechanism or dense decoder output for boxes is described; this is necessary for the perception reward to be computable.
  • domain assumption Rendering a 3D scene into six 2D views preserves the information required for reasoning, QA, and grounding.
    The dynamic view selection module assumes 2D multi-view images plus depth and point cloud are sufficient for all tasks.
  • domain assumption CLIP cosine similarity between predicted and ground-truth answers is a valid reward proxy for answer quality.
    The semantic similarity reward replaces human preference or task-specific metrics (e.g., CIDEr) with CLIP text embedding similarity; the paper does not validate that this proxy tracks the reported metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding." pith.science (2026). https://pith.science/paper/ERF45AGU

@misc{pith2026250723478,
  author       = {Pith},
  title        = {Pith review of: 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERF45AGU}},
  note         = {Machine review of arXiv:2507.23478}
}
read the original abstract

Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. However, current 3D VLMs often struggle with robust reasoning and generalization due to limitations in high-quality spatial data and the static nature of viewpoint assumptions. To address these challenges, we propose 3D-R1, a foundation model that enhances the reasoning capabilities of 3D VLMs. Specifically, we first construct a high-quality synthetic dataset with CoT, named Scene-30K, leveraging existing 3D-VL datasets and a data engine based on Gemini 2.5 Pro. It serves as cold-start initialization data for 3D-R1. Moreover, we leverage RLHF policy such as GRPO in the reinforcement learning training process to enhance reasoning capabilities and introduce three reward functions: a perception reward, a semantic similarity reward and a format reward to maintain detection accuracy and answer semantic precision. Furthermore, we introduce a dynamic view selection strategy that adaptively chooses the most informative perspectives for 3D scene understanding. Extensive experiments demonstrate that 3D-R1 delivers an average improvement of 10% across various 3D scene benchmarks, highlighting its effectiveness in enhancing reasoning and generalization in 3D scene understanding. Code: https://github.com/AIGeeksGroup/3D-R1. Website: https://aigeeksgroup.github.io/3D-R1.

Figures

Figures reproduced from arXiv: 2507.23478 by the authors.

Figure 1
Figure 1. 3D-R1 is an open-source generalist model that enhances the reasoning of 3D VLMs for unified scene understanding. Abstract Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. How￾ever, current 3D VLMs often struggle with robust reason￾ing and generalization due to limitations in high-quality sp… view at source ↗
Figure 2
Figure 2. (a) Architecture. It takes text, multi-view images, 3D point clouds, and depth maps as input and formulates com￾prehensive 3D tasks as autoregressive sequence prediction. (b) Distribution of question types. Scene-30K contains diverse categories. (c) Multi-task performance. 3D-R1 demonstrates strong performance across various tasks. (d) Generalizability. 3D-R1 exhibits remarkable generalizability with enhanced reason… view at source ↗
Figure 3
Figure 3. CoT data engine. The point cloud of a scene is first sent to scene dscription generator to get a description of the scene. Then based on the description, we apply Gemini 2.5 Pro to synthetic CoT data. The Proposed Method Overview The 3D-R1 framework unfolds in two main phases. In the first phase, we synthesize the Scene-30K dataset, which pairs 3D scenes with questions and coherent chains of thought (CoT). In the se… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: The pipeline of Reinforcement Learning based GRPO. The policy model generates N outputs from a point cloud and question. Then perception IoU, semantic CLIP-similarity, and format-adherence rewards are computed, grouped, and com￾bined with a KL term to a frozen referenc…
Figure 5
Figure 5. Figure 5: Performance surfaces under different dynamic view selection weight configurations. We analyze the influ￾ence of text relevance (wt), spatial coverage (wc), and CLIP￾based similarity (wclip) on model performance, with the con￾straint wc+wclip = 1. Results on 3D-QA (Scan…
Figure 6
Figure 6. Figure 6: Scene-30K CoT data example 1 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Scene-30K CoT data example 2 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Scene-30K CoT data example 3 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Scene-30K CoT data example 4 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Scene-30K CoT data example 5 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results for 3D scene dense captioning (3D-DC) [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Qualitative results for 3D object captioning [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Qualitative results for 3D visual grounding (3D-VG) [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Qualitative results for 3D question answering (3D-QA) [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Qualitative results for 3D dialogue [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Qualitative results for 3D reasoning [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Qualitative results for 3D planning [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Holo-Captioning: Toward the Text Equivalent of 3D Scenes

    cs.CV 2026-07 conditional novelty 6.5 of 10

    HoloScribe generates pure-text structured captions of all entities, boxes, attributes and relations in 3D indoor scenes and outperforms dense captioners and 3D LLMs on a new 15K-scene benchmark.

  2. VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Sparse multi-view reasoning is largely unsolved for VLMs; grounded CoT with visual evidence improves moderate cases and transfers to real data, but deep multi-hop reasoning still scales poorly.

  3. SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

    cs.CV 2026-08 reject novelty 6.0 of 10

    A depth-aware structured chain-of-thought trained with multi-objective process rewards improves VLM spatial reasoning benchmarks, though headline numbers are weakened by evaluation overlap and test-set tuning.

  4. RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Depth refinement for monocular 3D detection is reformulated as a visual alignment judgment made by a VLM emitting discrete direction and magnitude tokens, yielding consistent post-hoc gains without retraining the base...

  5. ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A geometry-consistent memory plus a self-supervised consistency-reinforcement stage improves video spatial reasoning by roughly 7–15 points on three benchmarks, with an advertised average gain of 12.6 points.

  6. Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Online geometry-aware voxel overlap pruning removes up to 50% of visual tokens from multi-view 3D scenes while improving zero-shot 3D QA on Qwen VL models.

  7. OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

    cs.CV 2025-12 conditional novelty 6.0 of 10

    OpenGround grounds open-world 3D targets by planning a task chain and dynamically expanding the object lookup table through online 2D segmentation and 3D lifting, achieving SOTA zero-shot ScanRefer accuracy and 46.2% ...

  8. PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding

    cs.CV 2025-12 conditional novelty 6.0 of 10

    PanoGrounder grounds natural-language queries in 3D scenes by rendering a few multi-modal 360° panoramas, asking a pretrained VLM for 2D boxes, and lifting them into one 3D box — reaching state-of-the-art on ScanRefer...

  9. VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery

    cs.CV 2026-07 conditional novelty 4.0 of 10

    VaseMuseum is a training-free multimodal agent that combines DeepResearch-style retrieval, source/response reliability control, and best-of-K reranking to improve citation validity and reduce hallucination for museum ...

  10. Nav-R1: Reasoning and Navigation in Embodied Scenes

    cs.RO 2025-09 reject novelty 4.0 of 10

    Nav-R1 uses a 110K synthetic CoT dataset, GRPO with three rewards, and a fast-in-slow system to set new SOTA on R2R-CE, RxR-CE, and HM3D-OVON.

  11. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

73 extracted references · 54 canonical work pages · cited by 11 Pith papers

  1. [1]

    Achlioptas, P.; Abdelreheem, A.; Xia, F.; Elhoseiny, M.; and Guibas, L. 2020. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, 422--440. Springer

    work page 2020

  2. [2]

    Azuma, D.; Miyanishi, T.; Kurita, S.; and Kawanabe, M. 2022. ScanQA: 3D Question Answering for Spatial Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2022

  3. [3]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  4. [4]

    M.; Alsaedy, Y

    BAKR, E. M.; Alsaedy, Y. Y.; and Elhoseiny, M. 2022. Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Grounding. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems

    work page 2022

  5. [5]

    Banerjee, S.; and Lavie, A. 2005. METEOR : An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Goldstein, J.; Lavie, A.; Lin, C.-Y.; and Voss, C., eds., Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , 65--72. Ann Arbor, Michigan: Association ...

    work page 2005

  6. [6]

    Cai, D.; Zhao, L.; Zhang, J.; Sheng, L.; and Xu, D. 2022. 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16464--16473

    work page 2022

  7. [7]

    Chang, C.-P.; Wang, S.; Pagani, A.; and Stricker, D. 2024. MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14131--14140

    work page 2024

  8. [8]

    Chen, D.; Gholami, A.; Niesner, M.; and Chang, A. 2021 a . Scan2Cap: Context-aware Dense Captioning in RGB-D Scans. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2021

Show all 73 references
  1. [9]

    Chen, D.; Hu, R.; Chen, X.; Nießner, M.; and Chang, A. 2023 a . UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 18063–18073

    2023

  2. [10]

    Z.; Chang, A

    Chen, D. Z.; Chang, A. X.; and Nie ner, M. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, 202--221. Springer

    2020

  3. [11]

    Z.; Wu, Q.; Nie ner, M.; and Chang, A

    Chen, D. Z.; Wu, Q.; Nie ner, M.; and Chang, A. X. 2021 b . D3Net: A Speaker-Listener Architecture for Semi-supervised Dense Captioning and Visual Grounding in RGB-D Scans. arXiv preprint arXiv:2112.01551

    2021 arXiv

  4. [12]

    Chen, S.; Chen, X.; Zhang, C.; Li, M.; Yu, G.; Fei, H.; Zhu, H.; Fan, J.; and Chen, T. 2024 a . LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning. In CVPR, 26418--26428

    2024

  5. [13]

    Chen, S.; Guhur, P.-L.; Tapaswi, M.; Schmid, C.; and Laptev, I. 2022. Language conditioned spatial relation reasoning for 3d object grounding. In NIPS

    2022

  6. [14]

    Chen, S.; Zhu, H.; Chen, X.; Lei, Y.; Yu, G.; and Chen, T. 2023 b . End-to-End 3D Dense Captioning with Vote2Cap-DETR. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11124–11133

    2023

  7. [15]

    Chen, S.; Zhu, H.; Li, M.; Chen, X.; Guo, P.; Lei, Y.; Yu, G.; Li, T.; and Chen, T. 2024 b . Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11): 7331–7347

    2024

  8. [16]

    Chen, Z.; Wu, J.; Lei, Z.; Pollefeys, M.; and Chen, C. W. 2025. Compile Scene Graphs with Reinforcement Learning. arXiv preprint arXiv:2504.13617

    2025 arXiv

  9. [17]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  10. [18]

    Deng, J.; He, T.; Jiang, L.; Wang, T.; Dayoub, F.; and Reid, I. 2025. 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2025

  11. [19]

    Feng, M.; Li, Z.; Li, Q.; Zhang, L.; Zhang, X.; Zhu, G.; Zhang, H.; Wang, Y.; and Mian, A. 2021. Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 3702--3711

    2021

  12. [20]

    Fu, R.; Liu, J.; Chen, X.; Nie, Y.; and Xiong, W. 2025. Scene-LLM: Extending Language Model for 3D Visual Reasoning. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2195--2206

    2025

  13. [21]

    P.; and Gool, L

    Halacheva, A.-M.; Zaech, J.-N.; Wang, X.; Paudel, D. P.; and Gool, L. V. 2025. GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond. arXiv preprint arXiv:2507.00886

    2025 arXiv

  14. [22]

    He, D.; Zhao, Y.; Luo, J.; Hui, T.; Huang, S.; Zhang, A.; and Liu, S. 2021. TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding. In Proceedings of the 29th ACM International Conference on Multimedia

    2021

  15. [23]

    Hong, Y.; Zhen, H.; Chen, P.; Zheng, S.; Du, Y.; Chen, Z.; and Gan, C. 2023. 3D- LLM : Injecting the 3D World into Large Language Models. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  16. [24]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  17. [25]

    Huang, H.; Chen, Y.; Wang, Z.; Huang, R.; Xu, R.; Wang, T.; Liu, L.; Cheng, X.; Zhao, Y.; Pang, J.; and Zhao, Z. 2024 a . Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  18. [26]

    Huang, J.; Yong, S.; Ma, X.; Linghu, X.; Li, P.; Wang, Y.; Li, Q.; Zhu, S.-C.; Jia, B.; and Huang, S. 2024 b . An Embodied Generalist Agent in 3D World. In ICLR 2024 Workshop: How Far Are We From AGI

    2024

  19. [27]

    Huang, P.-H.; Lee, H.-H.; Chen, H.-T.; and Liu, T.-L. 2021. Text-Guided Graph Neural Networks for Referring 3D Instance Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2): 1610--1618

    2021

  20. [28]

    Huang, S.; Chen, Y.; Jia, J.; and Wang, L. 2022. Multi-View Transformer for 3D Visual Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15524--15533

    2022

  21. [29]

    Huang, T.; Zhang, Z.; Wang, Y.; and Tang, H. 2025 a . 3D CoCa: Contrastive Learners are 3D Captioners. arXiv preprint arXiv:2504.09518

    2025 arXiv

  22. [30]

    Huang, T.; Zhang, Z.; Zhang, R.; and Zhao, Y. 2025 b . DC-Scene: Data-Centric Learning for 3D Scene Understanding. arXiv preprint arXiv:2505.15232

    2025 arXiv

  23. [31]

    Jain, A.; Gkanatsios, N.; Mediratta, I.; and Fragkiadaki, K. 2022. Bottom up top down detection transformers for language grounding in images and point clouds. In ECCV, 417--433. Springer

    2022

  24. [32]

    Jia, B.; Chen, Y.; Yu, H.; Wang, Y.; Niu, X.; Liu, T.; Li, Q.; and Huang, S. 2024. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision (ECCV)

    2024

  25. [33]

    Jiao, Y.; Chen, S.; Jie, Z.; Chen, J.; Ma, L.; and Jiang, Y.-G. 2022. MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenes. In In Proceedings of the European conference on computer vision, 528–545

    2022

  26. [34]

    Jin, Z.; Hayat, M.; Yang, Y.; Guo, Y.; and Lei, Y. 2023. Context-aware Alignment and Mutual Masking for 3D-Language Pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10984--10994

    2023

  27. [35]

    Kim, M.; Lim, H.; Lee, S.; Kim, B.; and Kim, G. 2025. Bi-directional Contextual Attention for 3D Dense Captioning. In In Proceedings of the European conference on computer vision, 385–401

    2025

  28. [36]

    Lichen, Z.; Daigang, C.; Lu, S.; and Dong, X. 2021. 3DVG-Transformer : Relation modeling for visual grounding on point clouds. In ICCV, 2928--2937

    2021

  29. [37]

    Lin, C.-Y. 2004. ROUGE : A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74--81. Barcelona, Spain: Association for Computational Linguistics

    2004

  30. [38]

    Luo, J.; Fu, J.; Kong, X.; Gao, C.; Ren, H.; Shen, H.; Xia, H.; and Liu, S. 2022. 3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection. arXiv preprint arXiv:2204.06272

    2022 arXiv

  31. [39]

    Luo, T.; Johnson, J.; and Lee, H. 2024. View selection for 3d captioning via diffusion ranking. In European Conference on Computer Vision, 180--197. Springer

    2024

  32. [40]

    Luo, T.; Rockwell, C.; Lee, H.; and Johnson, J. 2023. Scalable 3D Captioning with Pretrained Models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    2023

  33. [41]

    Ma, X.; Yong, S.; Zheng, Z.; Li, Q.; Liang, Y.; Zhu, S.-C.; and Huang, S. 2023. SQA3D: Situated Question Answering in 3D Scenes. In International Conference on Learning Representations

    2023

  34. [42]

    Mao, A.; Yang, Z.; Chen, W.; Yi, R.; and Liu, Y.-j. 2023. Complete 3D Relationships Extraction Modality Alignment Network for 3D Dense Captioning. IEEE Transactions on Visualization and Computer Graphics

    2023

  35. [43]

    Mo, W.; and Liu, Y. 2024. Bridging the gap between 2D and 3D visual question answering: a fusion approach for 3D VQA. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligenc...

    2024

  36. [44]

    OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; Avila, R.; Babuschkin, I.; Balaji, S.; Balcom, V.; Baltescu, P.; Bao, H.; Bavarian, M.; Belgum, J.; Bello, I.; Berdine, J.; Bernadett-Shapi...

    2024 arXiv

  37. [45]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, 311–318. USA: Association for Computational Linguistics

    2002

  38. [46]

    Parelli, M.; Delitzas, A.; Hars, N.; Vlassis, G.; Anagnostidis, S.; Bachmann, G.; and Hofmann, T. 2023. CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5606--5611

    2023

  39. [47]

    Park, S.; Kim, H.; Kim, J.; Kim, S.; and Ro, Y. M. 2025. DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes. arXiv preprint arXiv:2505.23179

    2025

  40. [48]

    T.; and Mildenhall, B

    Poole, B.; Jain, A.; Barron, J. T.; and Mildenhall, B. 2023. DreamFusion: Text-to-3D using 2D Diffusion. In ICLR

    2023

  41. [49]

    Qi, Z.; Zhang, Z.; Fang, Y.; Wang, J.; and Zhao, H. 2025. GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models. arXiv:2501.01428

    2025 arXiv

  42. [50]

    Shao, Z.; Wang, P.; ihao Zhu; Xu, R.; Song, J.; Zhang, M.; Y.K. Li, Y. W.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR, abs/2402.03300

    2024 arXiv

  43. [51]

    H.; Blukis, V.; Tremblay, J.; Tyree, S.; Su, Y.; and Birchfield, S

    Song, C. H.; Blukis, V.; Tremblay, J.; Tyree, S.; Su, Y.; and Birchfield, S. 2025. RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities

    2025

  44. [52]

    Tang, Y.; Han, X.; Li, X.; Yu, Q.; Hao, Y.; Hu, L.; and Chen, M. 2024. MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors. In Proceedings of the 32nd ACM International Conference on Multimedia, 6617–6626

    2024

  45. [53]

    Team, L.; Modi, A.; Veerubhotla, A. S.; Rysbek, A.; Huber, A.; Anand, A.; Bhoopchand, A.; Wiltshire, B.; Gillick, D.; Kasenberg, D.; Sgouritsa, E.; Elidan, G.; Liu, H.; Winnemoeller, H.; Jurenka, I.; Cohan, J.; She, J.; Wilkowski, J.; Alarakyia, K.; McKee, K. R.; Singh, K.; Wa...

    2025 arXiv

  46. [54]

    F.; Alabdulmohsin, I.; Parthasarathy, N.; Evans, T.; Beyer, L.; Xia, Y.; Mustafa, B.; Hénaff, O.; Harmsen, J.; Steiner, A.; and Zhai, X

    Tschannen, M.; Gritsenko, A.; Wang, X.; Naeem, M. F.; Alabdulmohsin, I.; Parthasarathy, N.; Evans, T.; Beyer, L.; Xia, Y.; Mustafa, B.; Hénaff, O.; Harmsen, J.; Steiner, A.; and Zhai, X. 2025. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding...

    2025 arXiv

  47. [55]

    L.; and Parikh, D

    Vedantam, R.; Zitnick, C. L.; and Parikh, D. 2015. CIDEr: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4566--4575

    2015

  48. [56]

    Wang, H.; Zhang, C.; Yu, J.; and Cai, W. 2022. Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 1393–1400

    2022

  49. [57]

    Wang, X.; Li, Z.; Xu, Y.; Qi, J.; Yang, Z.; Ma, R.; Liu, X.; and Zhang, C. 2025. Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models. arXiv preprint arXiv:2507.16524

    2025 arXiv

  50. [58]

    H.; Le, Q

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; brian ichter; Xia, F.; Chi, E. H.; Le, Q. V.; and Zhou, D. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Proc...

    2022

  51. [59]

    Wu, X.; Jiang, L.; Wang, P.-S.; Liu, Z.; Liu, X.; Qiao, Y.; Ouyang, W.; He, T.; and Zhao, H. 2024. Point Transformer V3: Simpler, Faster, Stronger. In CVPR

    2024

  52. [60]

    Wu, Y.; Cheng, X.; Zhang, R.; Cheng, Z.; and Zhang, J. 2023. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In CVPR, 19231--19242

    2023

  53. [61]

    Xu, R.; Wang, X.; Wang, T.; Chen, Y.; Pang, J.; and Lin, D. 2024. PointLLM: Empowering Large Language Models to Understand Point Clouds. In ECCV

    2024

  54. [62]

    Yang, L.; Kang, B.; Huang, Z.; Zhao, Z.; Xu, X.; Feng, J.; and Zhao, H. 2024. Depth Anything V2. arXiv preprint arXiv:2406.09414

    2024 arXiv

  55. [63]

    Yang, Z.; Zhang, S.; Wang, L.; and Luo, J. 2021. SAT: 2D Semantics Assisted Training for 3D Visual Grounding. In ICCV

    2021

  56. [64]

    Yuan, Z.; Jiang, S.; Feng, C.-M.; Zhang, Y.; Cui, S.; Li, Z.; and Zhao, N. 2025. Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations. arXiv preprint arXiv:2506.17545

    2025 arXiv

  57. [65]

    Yuan, Z.; Yan, X.; Liao, Y.; Zhang, R.; Li, Z.; and Cui, S. 2021. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer Visio...

    2021

  58. [66]

    Zhao, Y.; Lin, J.; Ye, S.; Pang, Q.; and Lau, R. W. 2024. OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding. arXiv preprint arXiv:2408.11030

    2024

  59. [67]

    Zheng, D.; Huang, S.; and Wang, L. 2025. Video-3D LLM : Learning Position-Aware Video Representation for 3D Scene Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2025

  60. [68]

    Zhi, H.; Chen, P.; Li, J.; Ma, S.; Sun, X.; Xiang, T.; Lei, Y.; Tan, M.; and Gan, C. 2024. LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences. arXiv preprint arXiv:2412.01292

    2024 arXiv

  61. [69]

    Zhong, Y.; Xu, L.; Luo, J.; and Ma, L. 2022. Contextual Modeling for 3D Dense Captioning on Point Clouds. arXiv preprint arXiv:2210.03925

    2022 arXiv

  62. [70]

    Zhu, C.; Wang, T.; Zhang, W.; Pang, J.; and Liu, X. 2024. LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness. CoRR, abs/2409.18125

    2024 arXiv

  63. [71]

    Zhu, Z.; Ma, X.; Chen, Y.; Deng, Z.; Huang, S.; and Li, Q. 2023. 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2911--2921

    2023

  64. [72]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  65. [73]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.