{"id":"a0b55fa0-436b-45fd-b33d-81255497c201","arxiv_id":"2506.17545","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Video-grounded RL with two-stage 2D grounding and SAM2 lifting enables 3D object localization and QA without dense 3D instance supervision.","lead":"Scene-R1 trains a video-grounded language model to locate objects in 3D scenes using only 2D box rewards and reinforcement learning, then lifts 2D masks into 3D point clouds. It reports competitive results on ScanRefer, SceneFun3D and SQA3D without point-wise 3D instance labels, while emitting step-by-step reasoning chains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never discloses how the ground-truth 2D boxes used as RL rewards in §4.2 are obtained; since ScanRefer provides no 2D boxes, they most plausibly come from projecting the 3D instance annotations the paper claims to avoid, undermining the central 'no 3D annotations' claim.","rationale":"The reader's weakest_assumption correctly identifies the undisclosed origin of the 2D reward boxes. This is the single load-bearing condition for the paper's central claim of annotation-free 3D grounding: without an independent 2D source of supervision, the method silently inherits the 3D instance labels it claims to avoid. The paper provides no evidence that such 2D boxes exist for ScanRefer, and the most natural way to obtain them is by projecting the 3D annotations. The concern is concrete and testable, so a conditional verdict is appropriate: if the test confirms 3D-derived boxes, the claim collapses; if it shows a genuine 2D source, the contribution stands. No ad hominem is intended; the issue is an omission in the method description. The internal numeric inconsistencies noted by the reader are real but separate; they affect credibility, not the logical structure of the claim. Therefore I agree with the reader and recommend no change to the CONDITIONAL verdict.","tokens_in":13732,"tokens_out":5496,"duration_ms":54392,"concrete_test":"Examine the released training code (or request it from the authors) and locate the data-preparation step that generates b*_tau for ScanRefer. If the script projects ScanNet ground-truth 3D instance boxes into each RGB frame using the camera pose to produce the 2D boxes and the visibility set L*, then the training rewards derive from 3D annotations and the central claim is falsified. If instead the boxes are obtained from a 2D source (e.g., manual 2D annotation or a 2D detector trained without 3D labels), the concern is resolved. As an alternative quantitative test, retrain the image-grounding stage using 2D boxes generated by a pretrained 2D grounding model (e.g., GroundingDINO) and compare final 3D grounding accuracy in Table 1; a large drop would indicate the supervision actually came from 3D-projected boxes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of Scene-R1 is that it achieves competitive 3D scene understanding 'without any point-wise 3D instance supervision' and 'without 3D annotations.' The training rewards in Section 4.1 and 4.2 depend on ground-truth data: the temporal reward rTG requires L*, the set of frames containing the target object, and the image reward rIG requires b*_tau, the ground-truth 2D box in every frame. Section 5.1 lists ScanRefer, SQA3D, and SceneFun3D as datasets, none of which provides 2D box annotations or per-frame visibility masks. The paper never states how b*_tau and L* are computed. The most plausible source is the ScanNet/ScanRefer 3D instance annotations: projecting the 3D object box into each frame using the known camera pose gives both the 2D box and the visibility set. If that is the case, the method is not free of 3D annotations; it merely converts them to a weaker 2D form. This would invalidate the abstract's 'without 3D Annotations' claim and the annotation-efficiency advantage over fully supervised systems, because the expensive 3D instance labels are still required to produce the rewards. The paper's own phrasing in Section 4.3 ('inexpensive 2D boxes') and the absence of any description of how these boxes are obtained in Section 5.1 make this concern concrete rather than speculative. The internal inconsistencies (e.g., temporal IoU in Figure 2) are secondary; this is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Scene-R1, a video-grounded VLM for 3D scene reasoning that replaces 3D detectors with a two-stage grounding pipeline. In Stage 1, a Qwen2.5-VL model selects query-relevant video snippets while generating chain-of-thought rationales; in Stage 2, it predicts per-frame 2D bounding boxes, which are fed to SAM2 to produce masks, and the masks are back-projected to 3D using depth and camera poses. Training uses GRPO with IoU-based and format rewards; for 3D VQA it uses exact-match rewards. Experiments on ScanRefer, SQA3D, and SceneFun3D report competitive results without dense point-wise 3D instance labels. The central claim is that this pipeline eliminates the need for 3D annotations while achieving performance comparable to detector-based methods.","tokens_in":14088,"tokens_out":6809,"duration_ms":70628,"significance":"If the central claim is supported, Scene-R1 would be a notable step: it would show that a generic vision-language model fine-tuned with lightweight reinforcement-learning rewards can perform 3D grounding, affordance grounding, and 3D VQA without a pretrained 3D detector, while exposing interpretable chains of thought. The pipeline is simple and in principle reproducible from public components (Qwen2.5-VL, GRPO, SAM2, RGB-D video). However, the significance hinges on the provenance of the 2D ground-truth boxes used as rewards; the paper must clarify this before the contribution can be assessed. The reported quantitative gains over label-free baselines are meaningful but are clouded by internal numerical inconsistencies.","major_comments":[{"comment":"The training rewards require ground-truth per-frame 2D boxes b*_tau and the ground-truth frame set L*, but Section 5.1 does not state how these are obtained for ScanRefer, SQA3D, or SceneFun3D. These datasets provide 3D instance or element annotations, not per-frame 2D boxes. If b*_tau and L* are produced by projecting 3D annotations with the known camera poses, then the pipeline uses exactly the 3D annotations that the abstract claims to avoid. Please disclose the exact source of these reward signals and, if they come from 3D projection, revise the title, abstract, and the 'supervision' column in Table 1, as the annotation-efficiency claim is load-bearing for the paper's main contribution.","section":"Section 4.2 (rIG) and Section 4.1 (rTG)"},{"comment":"The temporal IoU value of 0.71 contradicts the IoU formula in Section 4.1. With prediction 58-62 s and ground truth 52-66 s, the intersection is 4 s and the union is 14 s, so IoU = 0.29, not 0.71. Please correct the reported value or explain the alternative computation used.","section":"Figure 2"},{"comment":"The text reports SceneFun3D Acc@0.25/0.5 of 26.4/4.6 for the full model, but Table 4 lists 48.5/11.0. The zero-shot and SFT numbers quoted in the same sentence match the table (20.3/2.5 and 8.5/3.2), so the 26.4/4.6 values appear to be a typographical error; the 'more than doubling' statement is only true for the table values. Please reconcile the text and table.","section":"Section 5.4, Table 4"}],"minor_comments":[{"comment":"The text mentions 'ARKitScenes clips retain their native resolution', but ARKitScenes is not introduced as a dataset; clarify whether SceneFun3D is built on ARKitScenes or whether an additional dataset is used.","section":"Section 5.1"},{"comment":"References [34] and [35] are duplicate entries for the same SAM 2 paper; consolidate them.","section":"References"},{"comment":"Use consistent spelling of Qwen; the table and text use both 'Qwen2.5-VL' and 'QWen2.5-VL'.","section":"Throughout"},{"comment":"The word 'proporate' should be 'appropriate'.","section":"Appendix A"},{"comment":"The sentence 'each scan have a coresponding video' has grammar issues; it should be 'each scan has a corresponding video'.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The central question for the editor is whether the authors can justify the provenance of the 2D boxes used as rewards. The internal numeric inconsistencies (Figure 2, Table 4) suggest the manuscript needs careful revision. If the boxes come from projecting 3D labels, the paper's main claim is not supported and may warrant rejection after the authors' response."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the core idea is genuinely fresh: using R1-style RL on video with two grounding stages—temporal then image—and lifting 2D masks to 3D via SAM2 and depth, all without a pre-trained 3D detector. Second, the paper never says where the ground-truth 2D boxes used as RL rewards come from, and that omission threatens the headline claim.\n\nWhat it does well: it integrates existing pieces (Qwen2.5-VL, SAM2, GRPO) in a way that is more than an engineering hack. The two stages are cleanly separated, the reward design is simple (IoU plus format), and the qualitative examples show interpretable chain-of-thought. The results on ScanRefer and SQA3D are plausible and clearly behind fully supervised methods, which the limitations section acknowledges. The paper is readable and spans three tasks.\n\nThe soft spot is the annotation story. Section 4.2 defines an image reward using ground-truth 2D boxes b*_tau, and Section 4.1 uses a ground-truth frame set L*. The datasets used—ScanRefer, SQA3D, SceneFun3D—do not provide 2D boxes or per-frame visibility. The paper does not explain how these are obtained. The most obvious source is projecting the 3D instance annotations into each frame using known camera poses. If that is what the authors did, the phrase 'without 3D annotations' is misleading: the 3D labels are still required, just converted to 2D form. The abstract and Section 4.3 call these 'task-level 2D boxes' or 'inexpensive 2D boxes,' but the dataset section never grounds that claim. This is not a minor paperwork problem; it affects the main contribution. I'd want a clear statement of the annotation pipeline before trusting the results.\n\nSecondary issues are concrete and easy to fix: Figure 2 reports a temporal IoU of 0.71 for a 4-second intersection over a 14-second union, which is 0.29 by the paper's own formula, and Section 5.4 gives SceneFun3D numbers (26.4/4.6) that do not match Table 4 (48.5/11.0). These look like sloppy reporting, but they erode confidence.\n\nWho this is for: researchers working on 3D VLMs and annotation-efficient grounding. The paper has a promising idea but needs a major revision to clarify the training supervision and fix the internal inconsistencies.\n\nRecommendation: send it to peer review—the idea deserves scrutiny—but expect heavy revision. I would not cite it as 'without 3D annotations' until the reward box question is resolved.","headline":"The idea is fresh and the pipeline is clean, but the paper never accounts for where the training 2D boxes come from, which undercuts the 'no 3D annotations' claim in its current form.","tokens_in":14637,"tokens_out":3658,"would_cite":false,"duration_ms":32556,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scene-R1 claims that a single video-grounded VLM can do 3D grounding and VQA with no point-wise 3D labels.","keywords":["3D visual grounding","affordance grounding","3D question answering","reinforcement learning","chain-of-thought","video grounding","RGB-D","SAM2"],"falsifier":"Check the training pipeline of the image-grounding stage: if the ground-truth 2D box $b^\\star_\\tau$ in the IoU reward is obtained by projecting the ScanRefer or ScanNet 3D instance box into the frame, then training has used 3D instance supervision in disguised form, and the central claim of annotation-free 3D reasoning is falsified by that derivation.","tokens_in":13541,"feed_emoji":"🎯","tokens_out":7749,"duration_ms":71511,"temperature":0.7,"pith_summary":"Scene-R1 claims that a single vision-language model, trained only with reinforcement-learning rewards based on 2D boxes and text, can reason about 3D scenes from RGB-D video without ever seeing point-wise 3D instance labels. The model first selects the video snippets relevant to a natural-language query, then predicts a 2D bounding box on each selected frame, and finally uses a video segmenter plus depth and camera poses to lift the boxes into a 3D bounding box. This makes the decisions transparent: the model must emit a chain of thought before its answer. If the claim holds, expensive 3D annotation could be replaced by video plus weak 2D supervision for grounding, affordance, and question-answering tasks. The paper reports results competitive with detector-based pipelines on ScanRefer, SQA3D, and SceneFun3D.","feed_headline":"Video plus RL teaches a VLM to ground objects in 3D without 3D labels","feed_subtitle":"Two-stage reward pipeline selects snippets, predicts 2D boxes, lifts masks into 3D, and shows its reasoning","key_machinery":"The machinery is a two-stage grounding pipeline optimized end-to-end under group-relative policy optimization (GRPO), an RL update that normalizes rewards within a group of sampled responses and updates without a critic. Stage 1, temporal grounding, outputs a start-end time window for the query and is rewarded by temporal IoU against the set of frames containing the target object. Stage 2, image grounding, outputs a JSON 2D bounding box per retained frame and is rewarded by box IoU plus format and JSON-validity rewards. The boxes are then fed to SAM2 to get pixel masks, and each masked pixel is back-projected with depth and camera pose into a point cloud, whose tightest axis-aligned box is the final 3D answer. The reward structure carries the argument: GRPO's group-normalized, scale-invariant update lets heterogeneous rewards (frame IoU, box IoU, exact match) be mixed into a single learning signal.","core_discovery":"On its own terms, the central discovery is that R1-style reinforcement learning can turn a generic video-capable vision-language model into a 3D reasoner. With group-relative policy optimization and rewards that only measure overlap of predicted temporal windows and 2D boxes, the model learns to locate objects in 3D by back-projecting SAM2 masks produced from its own 2D predictions. The paper therefore claims that the 3D-detector stage that anchors existing 3D-aware large language models is not necessary: video, depth, and camera poses can carry the localization signal, provided the model is rewarded for visible reasoning and format compliance. It further claims that the same recipe works for 3D question answering by replacing IoU rewards with exact-match rewards on answers.","pith_inferences":["In my reading, the paper's 'without 3D annotations' claim is underdetermined: the per-frame ground-truth 2D boxes $b^\\star_\\tau$ used in the image-grounding reward are not part of the published ScanRefer or ScanNet data, and if they were projected from the 3D instance boxes, the supervision saving would be a relabeling rather than a removal.","A direct test that would extend the claim is to replace the ground-truth 2D boxes in the reward with boxes from a 2D detector or from SAM2's automatic masks; if grounding accuracy survives, the pipeline would genuinely scale to in-the-wild RGB-D video without any manual 3D annotation.","The approach's reliance on SAM2 propagation and on depth plus camera pose means deployment in dynamic or outdoor scenes is not a small step, and the paper itself lists outdoor and dynamic environments as future work."],"forward_implications":["3D visual grounding can be learned without a 3D detector or point-cloud instance masks, using only RGB-D video and 2D box rewards.","The same GRPO recipe transfers to 3D question answering by swapping IoU rewards for exact-match rewards on answers.","Training with RL and a visible thinking prompt beats both the zero-shot backbone and supervised fine-tuning on temporal and image grounding in the paper's ablations.","On the no-3D-supervision setting, the paper reports Acc@0.25/0.5 of 38.8/17.1 on ScanRefer, beating OpenScene by 25.6/10.6 points, and 12.0 AP25 on SceneFun3D affordance grounding."],"supporting_citations":[{"why":"Qwen2.5-VL backbone; supplies the pretrained 2D grounding and VQA capabilities that the RL stages build on.","marker":"[3]"},{"why":"DeepSeek-R1; the R1-style RL reasoning approach that motivates chain-of-thought via reinforcement learning.","marker":"[18]"},{"why":"Introduces GRPO, the group-relative policy optimization algorithm that carries all reward optimization in the paper.","marker":"[39]"},{"why":"SAM2; the prompt-based video segmenter that turns predicted 2D boxes into masks for back-projection to 3D.","marker":"[35]"},{"why":"ScanRefer; the primary 3D visual grounding benchmark and the source of the text queries and 3D metrics.","marker":"[6]"},{"why":"ScanNet; supplies the RGB-D reconstructions, depth, poses, and the 3D instance labels underlying ScanRefer evaluation.","marker":"[13]"},{"why":"SQA3D; defines the situated 3D question answering task and its exact-match metrics.","marker":"[30]"},{"why":"SceneFun3D; defines the task-driven affordance grounding benchmark and its AP metric.","marker":"[14]"},{"why":"OpenScene; the strongest label-free 3D baseline that Scene-R1 compares against on ScanRefer.","marker":"[32]"},{"why":"Mask3D; the detector with dense point-wise supervision that prior 3D-aware LLMs depend on and that Scene-R1 removes.","marker":"[38]"}],"fun_headline_variants":["RL alone lifts video LLMs to 3D scene reasoning without 3D labels","No 3D detectors: RL from 2D boxes gives LLMs 3D scene grounding","Video LLM learns 3D localization from 2D boxes via reinforcement learning","Scene-R1: reward-driven LLM reasons in 3D using video and no 3D data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that per-frame ground-truth 2D boxes for every training video are available without first deriving them from 3D instance labels; the paper never states where these boxes come from, and on ScanRefer-derived data such boxes would normally be projections of the very 3D annotations it claims to avoid.","fun_headline_variants_meta":{"raw":{"variants":["RL alone lifts video LLMs to 3D scene reasoning without 3D labels","No 3D detectors: RL from 2D boxes gives LLMs 3D scene grounding","Video LLM learns 3D localization from 2D boxes via reinforcement learning","Scene-R1: reward-driven LLM reasons in 3D using video and no 3D data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1551,"prompt_tokens":987,"completion_tokens":564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":467}},"tokens_in":603,"tokens_out":564,"duration_ms":5891,"temperature":1.0,"reasoning_tokens":467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:30:03.176391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the training pipeline of the image-grounding stage: if the ground-truth 2D box $b^\\star_\\tau$ in the IoU reward is obtained by projecting the ScanRefer or ScanNet 3D instance box into the frame, then training has used 3D instance supervision in disguised form, and the central claim of annotation-free 3D reasoning is falsified by that derivation.","supporting_citations":[{"cited_title":"Scannet: Richly-annotated 3d reconstructions of indoor scenes","cited_arxiv_id":null,"evidence_quote":"ScanNet; supplies the RGB-D reconstructions, depth, poses, and the 3D instance labels underlying ScanRefer evaluation."},{"cited_title":"Scenefun3d: fine-grained functionality and affordance understanding in 3d scenes","cited_arxiv_id":null,"evidence_quote":"SceneFun3D; defines the task-driven affordance grounding benchmark and its AP metric."},{"cited_title":"Openscene: 3d scene understanding with open vocabularies","cited_arxiv_id":null,"evidence_quote":"OpenScene; the strongest label-free 3D baseline that Scene-R1 compares against on ScanRefer."},{"cited_title":"Mask3d: Mask transformer for 3d semantic instance segmentation","cited_arxiv_id":null,"evidence_quote":"Mask3D; the detector with dense point-wise supervision that prior 3D-aware LLMs depend on and that Scene-R1 removes."}],"review_version":1}