{"id":"24546491-5279-4ba2-852f-32b028a67484","arxiv_id":"2608.12220","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A depth-aware structured chain-of-thought trained with multi-objective process rewards improves VLM spatial reasoning benchmarks, though headline numbers are weakened by evaluation overlap and test-set tuning.","lead":"SCOUT trains vision-language models to answer spatial questions by first listing object locations and depths, then reasoning from that layout. The authors report beating GPT-4o on several spatial benchmarks, but the evaluation has notable overlaps with the training data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/eval overlap: SCOUT-24k is built from EmbSpatial and CV-Bench-derived templates, yet the evaluation uses EmbSpatial and CV-Bench; no disjoint split is disclosed, so claimed gains over GPT-4o may be memorization.","rationale":"The central claim is an empirical one: a 7B open VLM surpasses GPT-4o on spatial benchmarks and transfers across formats. For that claim to be meaningful, the evaluation must be uncontaminated by training data. The paper's own Section 2.4 says SCOUT-24k is built from EmbSpatial and STVQA images with CV-Bench-adapted templates, and Section 3.1 evaluates on EmbSpatial, CV-Bench, and related benchmarks. Nowhere is a split or deduplication step described. Appendix A.2 mentions 'the EmbSpatial training set', which suggests a split exists, but the paper never confirms that this training set is disjoint from the evaluation split, nor that templates synthesized from CV-Bench do not reproduce evaluation questions. Without this disclosure, the reported gains are uninterpretable. The depth-estimator concern raised by the reader is real but secondary: a biased depth reward would affect the mechanism's interpretation, not necessarily the benchmark scores. The OOD contradiction is also real but only attacks the second half of the claim. The contamination issue is the one that can void the headline numerical comparisons entirely. A single concrete check, overlap analysis between SCOUT-24k and the evaluation instances, would settle it. If overlap is zero and the authors release the split, the paper deserves re-evaluation; as written, the omission justifies rejection.","tokens_in":17422,"tokens_out":8092,"duration_ms":66525,"concrete_test":"Compute instance-level overlap between SCOUT-24k and the evaluation sets: compare image hashes (e.g., perceptual hashes of EmbSpatial and CV-Bench images) and question-text similarity for all QA pairs in Tables 1-2. If any exact or near-duplicate (image, question) appears in both training and evaluation, rerun the evaluation on the non-overlapping subset; if the margin over GPT-4o shrinks substantially (e.g., below the reported 4.28%), the central claim is an artifact of benchmark contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 states that source images for SCOUT-24k are 'collected from EmbSpatial [14] and STVQA [4]', and that templates for reasoning are 'adapted from CV-Bench [44] and Spatial-SSRL [31]'. Section 3.1 then evaluates on EmbSpatial, CV-Bench, BLINK, RoboSpatial, SpatialBench, and 3DSRBench. The paper never states that training samples are disjoint from evaluation samples; Appendix A.2 refers to an 'EmbSpatial training set' but does not specify how evaluation instances were excluded from SCOUT-24k. For EmbSpatial and STVQA samples that 'already contain answer annotations', the pipeline 'reserve[s] them for reinforcement learning' (Section 2.4), meaning the RL reward is computed on the very benchmarks later used for the headline comparisons. If any training images or QA pairs coincide with evaluation items, the 4.28% margin over GPT-4o and the 16.85% improvement over baselines are not evidence for the proposed depth-aware CoT plus multi-objective process reward; they are evidence of train/test leakage. The omission of any split description is itself a correctness risk, since the strongest numerical claims depend entirely on the evaluation being uncontaminated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCOUT, a two-stage training pipeline for vision-language models that combines a structured chain-of-thought (caption, scene, analyze, answer) with multi-objective process rewards and token-level advantage estimation. The authors build a 24k-example dataset (SCOUT-24k) from EmbSpatial and STVQA images using Qwen-VL-Max annotations and Depth-Anything-3 depth estimates, cold-start with SFT, and then apply RL with rewards for grounding, depth, reasoning consistency, answer accuracy, and format. Experiments on six single-image benchmarks and two multi-image/video benchmarks report that SCOUT-3B and SCOUT-7B outperform Qwen2.5-VL baselines and that SCOUT-7B surpasses GPT-4o on the average of the single-image benchmarks, with additional gains on multi-image and video multiple-choice questions.","tokens_in":17655,"tokens_out":11779,"duration_ms":96946,"significance":"If the empirical claims were clean, SCOUT would be a useful contribution: the reward decomposition is concrete, the token-level advantage assignment in Eqs. (5)-(7) is a sensible way to address credit assignment in RLVR, and the ablation study in Table 4 attempts to isolate the contribution of each reward. The two major risks are (i) possible train/eval overlap because SCOUT-24k is built from EmbSpatial and CV-Bench-derived material while those benchmarks are used for evaluation, and (ii) test-set hyperparameter selection in Table 5. Both concerns are load-bearing for the headline comparisons, so the significance of the paper cannot be assessed until they are resolved.","major_comments":[{"comment":"SCOUT-24k is constructed from source images in EmbSpatial and STVQA (Section 2.4), and the evaluation in Section 3.1 uses EmbSpatial and CV-Bench, among others; the paper never states that the training/RL items are disjoint from the evaluation items. Appendix A.2 refers to an 'EmbSpatial training set' without defining its relationship to the EmbSpatial benchmark evaluation split, and Section 2.4 says that samples with existing answer annotations are 'reserved for reinforcement learning.' If any EmbSpatial images or QA pairs used in RL appear in the evaluation, the headline margins over GPT-4o (4.28%) and over baselines (16.85%) are not evidence for the proposed method but for train/test leakage. The authors must report exact image- and QA-level overlap statistics and re-run the evaluation on a strictly disjoint split, and clarify whether CV-Bench-derived templates or items were used for evaluation questions.","section":"§2.4, §3.1, Appendix A.2"},{"comment":"The mixing coefficients α1 and α2 in Eq. (6) are chosen by evaluating α∈{0.3,0.5,0.7} on the same six benchmarks that later constitute the main results (Table 5). Selecting α=0.3 as the best configuration on the evaluation set is test-set model selection; it invalidates the reported averages as unbiased estimates and inflates the apparent advantage of SCOUT. The sensitivity analysis should be conducted on a held-out validation split, and the test evaluation should be reported only for the final configuration.","section":"§3.4, Table 5"},{"comment":"The depth reward (Eq. 3) and the dataset's depth labels both treat Depth-Anything-3 estimates, sampled at the center of each bounding box, as ground truth. The paper provides no validation of this monocular estimator on the image distributions of the evaluation benchmarks. If the estimator is systematically biased, the RL reward teaches the model to reproduce the estimator's errors, so gains in depth-related columns (e.g., BLINK depth and 3D in Table 1) may reflect fitting to the estimator rather than genuine 3D understanding. Please quantify the estimator's error against available metric-depth ground truth in the benchmarks and, if possible, verify that the reward correlates with true depth.","section":"§2.4, Eq. (3)"}],"minor_comments":[{"comment":"The reported improvements of 16.85% and 6.3% are absolute percentage-point differences in average accuracy (e.g., Table 1: 60.71 to 77.56); please use 'percentage points' or report relative improvements.","section":"Abstract and §1"},{"comment":"The dataset 'STVQA [4]' is cited with reference [4], but reference [4] in the bibliography is Batra et al., SpatialThinker; no STVQA dataset entry is provided anywhere in the reference list.","section":"§2.4"},{"comment":"The z-score normalization divides by the group standard deviation σ_k, which is zero whenever all samples in a group have the same binary reward; please state how zero-variance groups are handled in the implementation.","section":"§2.3, Eq. (4)"},{"comment":"The claim of 'robust out-of-domain generalization' is weakened by the large drops on VSI-Bench numerical questions (25.03→19.26 for 3B and 40.52→29.35 for 7B); the text acknowledges this, but the abstract and Section 1 should qualify the generalization claim accordingly.","section":"Table 3"},{"comment":"The definitions of the 'BLINK', 'Overall 2D', and '3D' columns in Table 1 are not fully specified; please state which BLINK subset is used and how the average columns are computed.","section":"Appendix C.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a solid new assembly: a depth-aware structured CoT with multi-objective process rewards, segment-level advantage estimation, and a 24k dataset built to support it. The ablations are informative and the training dynamics in Fig. 4 match the claims. Second, the headline results are currently not trustworthy because the paper never shows that training data is disjoint from the evaluation benchmarks, and it tunes reward-mixing coefficients on the test benchmarks.\n\nThe plus side is real. The method is clearly described, the SFT-cold-start-plus-RL recipe is sensible, and the gains on benchmarks not used as training sources (RoboSpatial, 3DSRBench, BLINK) are suggestive that something authentic is happening. The body also honestly notes the VSI-Bench numerical question collapse, which the abstract glosses over with 'robust out-of-domain generalization.'\n\nThe soft spots, in order. (1) Train/eval overlap. SCOUT-24k uses EmbSpatial images and templates adapted from CV-Bench; evaluation includes EmbSpatial and CV-Bench. Appendix A.2 mentions an 'EmbSpatial training set,' which hints they used a train split, but the paper never explicitly says evaluation instances were excluded. For EmbSpatial samples with existing answer annotations, they are 'reserved for RL,' meaning the RL reward is computed on the same benchmark family later used for the headline comparisons. This is a load-bearing ambiguity. If the splits are clean, say so; if not, the 4.28% margin over GPT-4o and the 16.85% gain over baselines are not evidence. (2) Hyperparameter selection on test. Table 5 picks alpha based on the exact six benchmarks used for reporting. That is test-set tuning and contradicts the 'strict zero-shot evaluation' claim. The effect looks modest (67.94 vs 66.18), but it is still a methodological flaw. (3) The depth reward uses Depth-Anything-3 monocular estimates as ground truth. If that estimator is systematically biased, the model learns the bias. This is a limitation rather than a fatal flaw, but it deserves discussion. (4) No code, data, or checkpoints are released, which makes independent verification difficult.\n\nThis paper is for researchers working on RL-based spatial reasoning for VLMs and on benchmark evaluation practice. It deserves a serious referee; it is not a desk reject. But it needs major revision: explicit disjointness documentation or a re-run on clearly separate benchmarks, held-out hyperparameter selection, honest OOD reporting, and ideally released artifacts. As written, I would not accept the central claims.","headline":"A well-engineered spatial-reasoning RL recipe whose main numbers are currently not trustworthy because the paper never shows train/eval disjointness and tunes hyperparameters on the test benchmarks.","tokens_in":18239,"tokens_out":6504,"would_cite":false,"duration_ms":51907,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7-billion-parameter vision-language model, SCOUT-7B, surpasses GPT-4o on spatial reasoning benchmarks by training with structured, depth-aware chain-of-thought and multi-objective process rewards.","keywords":["spatial reasoning","vision-language models","reinforcement learning","process rewards","chain-of-thought","depth perception","3D scene understanding","credit assignment"],"falsifier":"Take held-out images with known 3D ground truth, such as RGB-D scans with metric depth, and compare SCOUT-7B's predicted <scene> depth values and relative-distance answers against both the true depths and Depth-Anything-3's estimates; if the model tracks the estimator's predictions but diverges from true metric depth on the same distribution, the reward has taught estimator-following rather than 3D understanding.","tokens_in":17173,"feed_emoji":"🧭","tokens_out":8768,"duration_ms":72298,"temperature":0.7,"pith_summary":"The paper aims to establish that the spatial-reasoning bottleneck of vision-language models can be broken by forcing the model to externalize 3D perception—bounding boxes plus depth—inside a structured chain of thought, then reinforcing each stage with its own reward rather than a single final-answer reward. It builds a synthesized spatial QA dataset, SCOUT-24k, and an RL algorithm that assigns token-level credit to scene perception, analysis, and the final answer. If the claims hold, a small open-weight model can outperform a proprietary frontier model on general and complex spatial benchmarks, and spatial skills learned on single images transfer to multi-image and video questions. The paper's central bet is that fine-grained process supervision, not raw scale, is what unlocks 3D reasoning in VLMs.","feed_headline":"7B model tops GPT-4o on spatial reasoning","feed_subtitle":"Step-by-step depth-aware rewards, not raw scale, push a small open vision model past a proprietary giant.","key_machinery":"The central mechanism is the tagged reasoning trace—<think><caption>…</caption><scene>…</scene><analyze>…</analyze></think><answer>…</answer>—which turns an otherwise opaque chain of thought into three separately verifiable modules. Bounding-box and depth values in the <scene> module make the model's 3D perception explicit, so RL can reward accurate object localization and depth with a Hungarian-matched grounding reward and a continuous depth reward. A blind verifier reward on <analyze> ensures the reasoning alone entails the answer, while the advantage estimator z-score normalizes the five rewards, blends local process advantages with the global outcome advantage at a weight of 0.3, and assigns those advantages token-wise by tag boundaries before the clipped policy update. This setup is what lets the model receive different gradient signals for perceiving, reasoning, and answering.","core_discovery":"On its own terms, the paper reports that SCOUT-7B, initialized from Qwen2.5-VL-7B and trained with an SFT cold-start followed by process-supervised RL, reaches 79.66% on the general spatial benchmark average, 4.28 points above GPT-4o, and 61.79% overall on complex spatial reasoning, 0.87 points above GPT-4o; the 3B version improves its base model by 16.85 points and 6.3 points on the two suites. The load-bearing components are a depth-aware structured CoT—<caption>, <scene> with bounding boxes and depths, <analyze>, <answer>—and a multi-objective process-reward RL with normalized rewards for grounding, depth, reasoning consistency, format, and accuracy, aggregated into scene, analysis, and outcome advantages and assigned to tokens by tag position. The paper also claims out-of-domain transfer from single-image training to ViewSpatial multi-image and VSI-Bench multiple-choice video questions, with gains of 2.46 and 3.13 points over the base model. Ablations show that removing the perception advantage stops the grounding and depth rewards from improving, and removing the analysis advantage collapses the consistency reward and drops RoboSpatial accuracy, supporting the claim that fine-grained credit assignment is what makes the pipeline work.","pith_inferences":["Editorial inference: since the depth reward and dataset labels both come from a single monocular depth estimator, the method's ceiling is tied to that estimator's accuracy; training with multiple depth predictors or metric-depth supervision would reveal whether the gains are genuine 3D knowledge or estimator imitation.","Editorial inference: the token-level advantage decomposition suggests a general recipe for other structured visual outputs—declare a verifiable grammar, reward each module, then blend local and global advantages—which could transfer to detection, planning, or embodied question answering.","Editorial inference: because the method requires a strictly structured CoT format, the same RL machinery may underperform on free-form reasoning; a natural test is to relax the tags after training and measure how much spatial accuracy survives.","Editorial inference: the paper's own VSI-Bench numerical-question results drop sharply, which suggests temporal and absolute quantitative estimation is not transferred; adding video-depth or tracking pseudo-labels to the reward set would be a targeted extension beyond the paper's single-image scope."],"forward_implications":["If the central claim is right, a compact open-weight model can beat a proprietary frontier model on spatial benchmarks, so frontier-level spatial reasoning no longer requires closed systems.","The ablations imply that outcome-only RL or vanilla CoT leaves large gains on the table: the full method's 67.94% average beats GRPO with vanilla CoT by 4.04 points and beats process-free or credit-free variants by about 2.7 points.","The observed transfer from single-image training to multi-image and video tasks suggests the model learns abstract 3D structure rather than image-format-specific shortcuts, at least for relative-direction and multiple-choice video questions.","Because gains appear in both general and complex benchmarks, the depth-aware structured format plus process rewards appears to improve perception and reasoning jointly, not just format following."],"supporting_citations":[{"why":"Supplies the base VLM architectures (Qwen2.5-VL-3B/7B) that SCOUT initializes from, as well as Qwen-VL-Max used in the data synthesis pipeline.","marker":"[3]"},{"why":"Provides the monocular depth estimates that serve as depth ground truth in dataset construction and as the target of the depth reward.","marker":"[27]"},{"why":"Introduces GRPO, the group-relative policy optimization baseline that SCOUT extends with multi-objective process rewards and token-level advantage assignment.","marker":"[40]"},{"why":"The Hungarian matching algorithm aligns predicted objects with ground-truth objects, forming the basis of the grounding and depth rewards.","marker":"[24]"},{"why":"Provides STVQA source images with bounding-box annotations and represents the structured-CoT baseline that SCOUT builds on and compares against.","marker":"[4]"},{"why":"EmbSpatial supplies the other source dataset with spatial QA annotations and bounding boxes used to construct SCOUT-24k.","marker":"[14]"},{"why":"Defines the Efficient IoU term used in the matching cost and quality term of the regularized grounding reward.","marker":"[54]"},{"why":"GPT-4o is the proprietary frontier model that SCOUT-7B is claimed to outperform on both general and complex spatial benchmark suites.","marker":"[23]"}],"fun_headline_variants":["Small 7B model tops GPT-4o on spatial reasoning via structured CoT","Depth-aware process rewards lift 7B SCOUT past GPT-4o","SCOUT-7B: fine-grained credit assignment beats scale in spatial tasks","Structured depth CoT + multi-objective RL: 7B outperforms GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pipeline assumes that the monocular depth estimates from Depth-Anything-3, sampled at each bounding box's center, are accurate enough to serve as ground truth for the dataset and as the RL depth reward, so any systematic bias in that estimator is what the model is trained to reproduce.","fun_headline_variants_meta":{"raw":{"variants":["Small 7B model tops GPT-4o on spatial reasoning via structured CoT","Depth-aware process rewards lift 7B SCOUT past GPT-4o","SCOUT-7B: fine-grained credit assignment beats scale in spatial tasks","Structured depth CoT + multi-objective RL: 7B outperforms GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1699,"prompt_tokens":1093,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":515}},"tokens_in":709,"tokens_out":606,"duration_ms":5063,"temperature":1.0,"reasoning_tokens":515,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:12:12.173427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take held-out images with known 3D ground truth, such as RGB-D scans with metric depth, and compare SCOUT-7B's predicted <scene> depth values and relative-distance answers against both the true depths and Depth-Anything-3's estimates; if the model tracks the estimator's predictions but diverges from true metric depth on the same distribution, the reward has taught estimator-following rather than 3D understanding.","supporting_citations":[{"cited_title":"Naval research logistics quarterly2(1-2), 83–97 (1955) 5","cited_arxiv_id":null,"evidence_quote":"The Hungarian matching algorithm aligns predicted objects with ground-truth objects, forming the basis of the grounding and depth rewards."},{"cited_title":"In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","cited_arxiv_id":null,"evidence_quote":"EmbSpatial supplies the other source dataset with spatial QA annotations and bounding boxes used to construct SCOUT-24k."},{"cited_title":"Neurocomputing506, 146–157 (2022) 5","cited_arxiv_id":null,"evidence_quote":"Defines the Efficient IoU term used in the matching cost and quality term of the regularized grounding reward."}],"review_version":1}