Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scene-R1 claims that a single video-grounded VLM can do 3D grounding and VQA with no point-wise 3D labels.

desk verdict The idea is fresh and the pipeline is clean, but the paper never accounts for where the training 2D boxes come from, which undercuts the 'no 3D annotations' claim in its current form. read the letter →

arxiv 2506.17545 v1 pith:3BE4J2GE submitted 2025-06-21 cs.CV

Zhihao Yuan , Shuyi Jiang , Chun-Mei Feng , Yaolun Zhang , Shuguang Cui , Zhen Li , Na Zhao This is my paper · ORCID
classification cs.CV
keywords 3Dvisualgroundingaffordancequestionansweringreinforcementlearningchain-of-thoughtvideoRGB-DSAM2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Scene-R1 claims that a single vision-language model, trained only with reinforcement-learning rewards based on 2D boxes and text, can reason about 3D scenes from RGB-D video without ever seeing point-wise 3D instance labels. The model first selects the video snippets relevant to a natural-language query, then predicts a 2D bounding box on each selected frame, and finally uses a video segmenter plus depth and camera poses to lift the boxes into a 3D bounding box. This makes the decisions transparent: the model must emit a chain of thought before its answer. If the claim holds, expensive 3D annotation could be replaced by video plus weak 2D supervision for grounding, affordance, and question-answering tasks. The paper reports results competitive with detector-based pipelines on ScanRefer, SQA3D, and SceneFun3D.

What carries the argument

The machinery is a two-stage grounding pipeline optimized end-to-end under group-relative policy optimization (GRPO), an RL update that normalizes rewards within a group of sampled responses and updates without a critic. Stage 1, temporal grounding, outputs a start-end time window for the query and is rewarded by temporal IoU against the set of frames containing the target object. Stage 2, image grounding, outputs a JSON 2D bounding box per retained frame and is rewarded by box IoU plus format and JSON-validity rewards. The boxes are then fed to SAM2 to get pixel masks, and each masked pixel is back-projected with depth and camera pose into a point cloud, whose tightest axis-aligned box is the final 3D answer. The reward structure carries the argument: GRPO's group-normalized, scale-invariant update lets heterogeneous rewards (frame IoU, box IoU, exact match) be mixed into a single learning signal.

What would settle it

Check the training pipeline of the image-grounding stage: if the ground-truth 2D box $b^\star_\tau$ in the IoU reward is obtained by projecting the ScanRefer or ScanNet 3D instance box into the frame, then training has used 3D instance supervision in disguised form, and the central claim of annotation-free 3D reasoning is falsified by that derivation.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that R1-style reinforcement learning can turn a generic video-capable vision-language model into a 3D reasoner. With group-relative policy optimization and rewards that only measure overlap of predicted temporal windows and 2D boxes, the model learns to locate objects in 3D by back-projecting SAM2 masks produced from its own 2D predictions. The paper therefore claims that the 3D-detector stage that anchors existing 3D-aware large language models is not necessary: video, depth, and camera poses can carry the localization signal, provided the model is rewarded for visible reasoning and format compliance. It further claims that the same recipe works for 3D question answering by replacing IoU rewards with exact-match rewards on answers.

Load-bearing premise

The load-bearing premise is that per-frame ground-truth 2D boxes for every training video are available without first deriving them from 3D instance labels; the paper never states where these boxes come from, and on ScanRefer-derived data such boxes would normally be projections of the very 3D annotations it claims to avoid.

Editorial extensions

If this is right

  • 3D visual grounding can be learned without a 3D detector or point-cloud instance masks, using only RGB-D video and 2D box rewards.
  • The same GRPO recipe transfers to 3D question answering by swapping IoU rewards for exact-match rewards on answers.
  • Training with RL and a visible thinking prompt beats both the zero-shot backbone and supervised fine-tuning on temporal and image grounding in the paper's ablations.
  • On the no-3D-supervision setting, the paper reports Acc@0.25/0.5 of 38.8/17.1 on ScanRefer, beating OpenScene by 25.6/10.6 points, and 12.0 AP25 on SceneFun3D affordance grounding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In my reading, the paper's 'without 3D annotations' claim is underdetermined: the per-frame ground-truth 2D boxes $b^\star_\tau$ used in the image-grounding reward are not part of the published ScanRefer or ScanNet data, and if they were projected from the 3D instance boxes, the supervision saving would be a relabeling rather than a removal.
  • A direct test that would extend the claim is to replace the ground-truth 2D boxes in the reward with boxes from a 2D detector or from SAM2's automatic masks; if grounding accuracy survives, the pipeline would genuinely scale to in-the-wild RGB-D video without any manual 3D annotation.
  • The approach's reliance on SAM2 propagation and on depth plus camera pose means deployment in dynamic or outdoor scenes is not a small step, and the paper itself lists outdoor and dynamic environments as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Scene-R1, a video-grounded VLM for 3D scene reasoning that replaces 3D detectors with a two-stage grounding pipeline. In Stage 1, a Qwen2.5-VL model selects query-relevant video snippets while generating chain-of-thought rationales; in Stage 2, it predicts per-frame 2D bounding boxes, which are fed to SAM2 to produce masks, and the masks are back-projected to 3D using depth and camera poses. Training uses GRPO with IoU-based and format rewards; for 3D VQA it uses exact-match rewards. Experiments on ScanRefer, SQA3D, and SceneFun3D report competitive results without dense point-wise 3D instance labels. The central claim is that this pipeline eliminates the need for 3D annotations while achieving performance comparable to detector-based methods.

Significance. If the central claim is supported, Scene-R1 would be a notable step: it would show that a generic vision-language model fine-tuned with lightweight reinforcement-learning rewards can perform 3D grounding, affordance grounding, and 3D VQA without a pretrained 3D detector, while exposing interpretable chains of thought. The pipeline is simple and in principle reproducible from public components (Qwen2.5-VL, GRPO, SAM2, RGB-D video). However, the significance hinges on the provenance of the 2D ground-truth boxes used as rewards; the paper must clarify this before the contribution can be assessed. The reported quantitative gains over label-free baselines are meaningful but are clouded by internal numerical inconsistencies.

major comments (3)
  1. [Section 4.2 (rIG) and Section 4.1 (rTG)] The training rewards require ground-truth per-frame 2D boxes b*_tau and the ground-truth frame set L*, but Section 5.1 does not state how these are obtained for ScanRefer, SQA3D, or SceneFun3D. These datasets provide 3D instance or element annotations, not per-frame 2D boxes. If b*_tau and L* are produced by projecting 3D annotations with the known camera poses, then the pipeline uses exactly the 3D annotations that the abstract claims to avoid. Please disclose the exact source of these reward signals and, if they come from 3D projection, revise the title, abstract, and the 'supervision' column in Table 1, as the annotation-efficiency claim is load-bearing for the paper's main contribution.
  2. [Figure 2] The temporal IoU value of 0.71 contradicts the IoU formula in Section 4.1. With prediction 58-62 s and ground truth 52-66 s, the intersection is 4 s and the union is 14 s, so IoU = 0.29, not 0.71. Please correct the reported value or explain the alternative computation used.
  3. [Section 5.4, Table 4] The text reports SceneFun3D Acc@0.25/0.5 of 26.4/4.6 for the full model, but Table 4 lists 48.5/11.0. The zero-shot and SFT numbers quoted in the same sentence match the table (20.3/2.5 and 8.5/3.2), so the 26.4/4.6 values appear to be a typographical error; the 'more than doubling' statement is only true for the table values. Please reconcile the text and table.
minor comments (5)
  1. [Section 5.1] The text mentions 'ARKitScenes clips retain their native resolution', but ARKitScenes is not introduced as a dataset; clarify whether SceneFun3D is built on ARKitScenes or whether an additional dataset is used.
  2. [References] References [34] and [35] are duplicate entries for the same SAM 2 paper; consolidate them.
  3. [Throughout] Use consistent spelling of Qwen; the table and text use both 'Qwen2.5-VL' and 'QWen2.5-VL'.
  4. [Appendix A] The word 'proporate' should be 'appropriate'.
  5. [Section 4] The sentence 'each scan have a coresponding video' has grammar issues; it should be 'each scan has a corresponding video'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RL rewards are external targets and the 3D output is a fixed back-projection of predicted boxes; the unstated origin of the 2D box rewards is a correctness risk, not a circular reduction.

full rationale

The paper's derivation chain is self-contained with respect to the circularity definitions checked here. The rewards in Sections 4.1 and 4.2, rTG = IoUtime(L^, L*) + lambda*rform and rIG = IoUbox(b^_tau, b*_tau) + lambda*(rjson + rform), are external supervised targets rather than functions of the model's own output, and GRPO is applied to these targets in the standard way. The lifting from 2D boxes to 3D in Section 4.3 is a fixed geometric operation (SAM2 mask, back-projection via intrinsic matrix K, world transform T_tau, then axis-aligned bounding box) with no learned parameters and no equation that reads the answer off the input. I therefore find no step where a 'prediction' is equal to its input by construction, and no load-bearing self-citation chain: the self-citations in Related Work are contextual descriptions of prior methods, not justifications of the current premise. The one substantive concern is an omitted provenance rather than circularity: ScanRefer, SQA3D, and SceneFun3D as described in Section 5.1 provide 3D instance annotations and language queries but are not stated to provide the ground-truth 2D boxes b*_tau or per-frame visibility sets L* required by the rewards. If those are generated by projecting 3D instance boxes into RGB frames, then the headline claim of training 'without 3D annotations' would be unsupported, and the annotation-efficiency advantage would be weakened. That is a supervision-provenance gap to verify, but it does not make the trained prediction equivalent to its training input; the model must still map RGB-D video to 2D boxes at inference, and the final 3D result is not copied from any stored label. Because no reduction, fitted-input renaming, or imported uniqueness theorem is exhibited, the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its free parameters are limited to the format-reward weight lambda. The key unstated assumption is the availability of 2D boxes, which likely derive from 3D annotations, and the reliance on SAM2 and camera calibration for accurate lifting.

free parameters (1)
  • lambda (reward weight) = 0.1
    Weight for the format reward in all RL objectives (rTG, rIG, rQA). Fixed by hand, not fit to data.
assumptions (3)
  • ad hoc to paper Ground-truth 2D boxes are available for every video frame during training.
    The paper uses b*_tau for IoU rewards in Section 4.2 but does not state how they are obtained. In ScanRefer they must be projected from 3D boxes, conflicting with the 'no 3D annotations' claim.
  • domain assumption SAM2 can segment the target object from a 2D box prompt accurately enough for 3D lifting.
    The 3D output inherits SAM2 errors; no analysis of segmentation failure is provided.
  • domain assumption Camera intrinsics and poses are known and correct for all frames.
    The lifting step projects pixels via K and T_tau; errors directly bias the 3D box.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations." pith.science (2026). https://pith.science/paper/3BE4J2GE

@misc{pith2026250617545,
  author       = {Pith},
  title        = {Pith review of: Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BE4J2GE}},
  note         = {Machine review of arXiv:2506.17545}
}
read the original abstract

Currently, utilizing large language models to understand the 3D world is becoming popular. Yet existing 3D-aware LLMs act as black boxes: they output bounding boxes or textual answers without revealing how those decisions are made, and they still rely on pre-trained 3D detectors to supply object proposals. We introduce Scene-R1, a video-grounded framework that learns to reason about 3D scenes without any point-wise 3D instance supervision by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline. In the temporal grounding stage, we explicitly reason about the video and select the video snippets most relevant to an open-ended query. In the subsequent image grounding stage, we analyze the image and predict the 2D bounding box. After that, we track the object using SAM2 to produce pixel-accurate masks in RGB frames, and project them back into 3D, thereby eliminating the need for 3D detector-based proposals while capturing fine geometry and material cues. Scene-R1 can also adapt to the 3D visual question answering task to answer free-form questions directly from video. Our training pipeline only needs task-level 2D boxes or textual labels without dense 3D point-wise labels. Scene-R1 surpasses existing open-vocabulary baselines on multiple datasets, while delivering transparent, step-by-step rationales. These results show that reinforcement-learning-based reasoning combined with RGB-D video alone offers a practical, annotation-efficient route to trustworthy 3D scene understanding.

Figures

Figures reproduced from arXiv: 2506.17545 by the authors.

Figure 1
Figure 1. Overview of Scene-R1 versus prior 3D LLMs. (a) Previous 3D LLMs directly output predictions as black boxes without exposing intermediate reasoning, while still depending on pre￾trained 3D encoders. (b) In contrast, Scene-R1 performs transparent 3D scene reasoning without any 3D instance supervision. think aloud, producing detailed chains of thought and higher task accuracy without human-written rationales. Vision-R1… view at source ↗
Figure 2
Figure 2. Visualization of visual grounding. 3D Question Answering [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualization of task-driven affordance grounding. For each instruction, we show the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of 3D visual grounding. Scene-R1 thinking: To determine what is on the 12 o'clock of the chair that is on my right, I need to consider the orientation of the room and the relative positions of the objects. Since the door is to my right in the other side o…
Figure 5
Figure 5. Figure 5: Visualization of 3D visual question answering. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5 of 10

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  2. 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    3D-R1 uses a synthetic chain-of-thought cold start plus GRPO reinforcement learning with perception, semantic, and format rewards, and reports best published results across 3D dense captioning, QA, grounding, dialogue...

Reference graph

Works this paper leans on

51 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, pages 422–440. Springer, 2020

    work page 2020

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022

    2022

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021

    arXiv 2021

  5. [5]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287–318. PMLR, 2023

    2023

  6. [6]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202–221. Springer, 2020

    2020

  7. [7]

    R1-v: Reinforcing super generalization ability in vision-language models with less than $3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https://github.com/Deep-Agent/R1-V , 2025. Accessed: 2025-02-02

    2025

  8. [8]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in Neural Information Processing Systems, 35:20522–20535, 2022

    work page 2022

Show all 51 references
  1. [9]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11124–11133, 2023

    2023

  2. [10]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, YU Gang, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  3. [11]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3193–3203, 2021

    2021

  4. [12]

    Functionality understanding and segmentation in 3d scenes

    Jaime Corsetti, Francesco Giuliari, Alice Fasoli, Davide Boscaini, and Fabio Poiesi. Functionality understanding and segmentation in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  5. [13]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828–5839, 2017

    2017

  6. [14]

    Scenefun3d: fine-grained functionality and affordance understanding in 3d scenes

    Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. Scenefun3d: fine-grained functionality and affordance understanding in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  7. [15]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  8. [16]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  9. [17]

    The ecological approach to visual perception: classic edition

    James J Gibson. The ecological approach to visual perception: classic edition. Psychology press, 2014

    2014

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    Viewrefer: Grasp the multi-view knowledge for 3d visual grounding

    Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15372–15383, 2023

    2023

  12. [20]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arX...

    2023 arXiv

  13. [21]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference on Neural Information Pr...

    2024

  14. [22]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023

    2023 arXiv

  15. [23]

    Text-guided graph neural networks for referring 3d instance segmentation

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. 35th AAAI Conference on Artificial Intelligence, 2021

    2021

  16. [24]

    Multi-view transformer for 3d visual grounding

    Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524– 15533, 2022

    2022

  17. [25]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  18. [26]

    Bottom up top down detection transformers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings,...

    2022

  19. [27]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739, 2023

    2023

  20. [28]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  21. [29]

    Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding

    Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. Seeground: See and ground for zero-shot open-vocabulary 3d visual grounding. arXiv preprint arXiv:2412.04383, 2024

    2024 arXiv

  22. [30]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. InInternational Conference on Learning Representations, 2023

    2023

  23. [31]

    Introducing openai o1

    OpenAI. Introducing openai o1. https://openai.com/o1/, 2024. Accessed: 2025-05-15

    2024

  24. [32]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 815–824, 2023

    2023

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  26. [35]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  27. [36]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    2021

  28. [37]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  29. [38]

    Mask3d: Mask transformer for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023

    2023

  30. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  31. [40]

    Chatgpt for robotics: Design principles and model abilities

    Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023

    2023

  32. [41]

    Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024

    arXiv

  33. [42]

    Timezero: Temporal video grounding with reasoning-guided lvlm

    Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. Timezero: Temporal video grounding with reasoning-guided lvlm. arXiv preprint arXiv:2503.13377, 2025

    2025 arXiv

  34. [43]

    Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding

    Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pag...

    2023

  35. [44]

    Pointllm: Empower- ing large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In ECCV, 2024

    2024

  36. [45]

    Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Com...

    2021

  37. [46]

    Visual programming for zero-shot open-vocabulary 3d visual grounding

    Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open-vocabulary 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623–20633, 2024

    2024

  38. [47]

    Empowering large language models with 3d situation awareness

    Zhihao Yuan, Yibo Peng, Jinke Ren, Yinghong Liao, Yatong Han, Chun-Mei Feng, Hengshuang Zhao, Guanbin Li, Shuguang Cui, and Zhen Li. Empowering large language models with 3d situation awareness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19...

    2025

  39. [48]

    3dvg-transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937, 2021

    2021

  40. [49]

    Bubogpt: Enabling visual grounding in multi-modal llms

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023

    2023 arXiv

  41. [50]

    Uni3d: Exploring unified 3d representation at scale

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773, 2023

    2023 arXiv

  42. [51]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023

    2023

  43. [52]

    [EVENT]" in the video, determine the precise time period of the occurrence of the object. Provide the start and end times (in seconds, precise to one decimal place) in the format

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. In European Conference on Computer Vision, pages 188–206. Springer, 2024. 12 A Prompts To pro...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.