{"id":"64271978-ea4b-48f5-a482-6c3632d35e4c","arxiv_id":"2505.12312","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 7B model trained on a new 322K indoor spatial QA dataset achieves the best average VSI-Bench score, but not on all eight tasks as claimed.","lead":"The authors built a 322,000-question dataset from indoor videos and fine-tuned a 7-billion-parameter vision-language model, reporting large gains on the VSI-Bench spatial reasoning benchmark. The headline claim that it beats all models on all eight tasks is not supported by the paper's own table for two of the tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark contamination is not ruled out, and the 'all eight tasks' claim is contradicted by Table 1 itself.","rationale":"The core contribution is empirical, so its validity hinges on two checkable points: the exact wording of the 'all eight' claim and the decontamination of the training data. Table 1 independently disproves the 'all eight' wording, since Gemini-1.5 Pro outperforms ViCA-7B on Relative Direction and Route Planning. More importantly, the paper's filtering procedure is unspecified, and both training and evaluation draw from the same three scene datasets with similar answer formats. This raises a concrete risk that the large Absolute Distance improvement comes from source-level overlap or format memorization rather than spatial reasoning. The reader's weakest assumption identified the same load-bearing issue, and the correct resolution is to require the authors to release the filtering artifacts and reproduce the result on genuinely held-out scenes. Until such evidence is provided, the claim should be treated as conditional, not rejected, because the paper has released code and results that could substantiate the finding with modest additional documentation.","tokens_in":89,"tokens_out":3697,"duration_ms":52233,"concrete_test":"Obtain the exact deduplication script and a list of VSI-Bench video/scan IDs excluded from ViCA-322K. Then evaluate ViCA-7B on the subset of VSI-Bench videos whose scene IDs were not seen during training; if no such videos remain, construct a held-out set of ARKitScenes/ScanNet/ScanNet++ scenes with VSI-Bench-style questions. If the reported 57.0 Absolute Distance or 60.6 average drops substantially on this held-out subset, the claimed generalization is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—ViCA-7B achieves SOTA on all eight VSI-Bench tasks—is not supported. First, Table 1 itself shows Gemini-1.5 Pro scores 46.3 vs ViCA-7B's 42.6 on Rel. Dir. and 36.0 vs 34.5 on Route Plan; the paper even computes its margins for these tasks relative to 7B/8B models, not the true best. Second, and more serious, §3.1 says the authors 'exclusively use the training splits of these datasets and conduct strict filtering' against VSI-Bench overlap, but provides no procedure, no video-ID lists, and no deduplication method. VSI-Bench is built from the same three datasets (ARKitScenes, ScanNet, ScanNet++), and the ViCA training QA formats mirror the benchmark's numeric and multiple-choice formats. If any scan IDs used in VSI-Bench appear in ViCA-322K, or if the model learns scene-specific answer distributions (e.g., common absolute distances), the +26.1 Absolute Distance gap (57.0 vs 30.9) could reflect memorization rather than generalized spatial cognition. This cannot be checked from the paper as written, so the headline result is not yet verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ViCA-322K, a 322,003-pair video QA dataset built from the training splits of ARKitScenes, ScanNet, and ScanNet++, covering six metadata-grounded 'Base Data' tasks and a 'Complex Spatial Reasoning' subset generated with GPT-4o-mini. The authors fine-tune LLaVA-Video-7B-Qwen2 on this dataset to obtain ViCA-7B, and report VSI-Bench results with an average score of 60.6, including a 26.1-point gain on Absolute Distance over prior models. They also construct ViCA-Thinking-2.68K from Gemini 2.5 Pro outputs, fine-tune ViCA-7B-Thinking, and show that structured reasoning improves interpretability at a substantial accuracy cost. The abstract and introduction claim new state-of-the-art performance on all eight VSI-Bench tasks, and the conclusion frames the gains as evidence that targeted spatial supervision transfers to held-out spatial reasoning tasks.","tokens_in":29705,"tokens_out":8005,"duration_ms":75578,"significance":"If the headline claims were fully supported, the contribution would be significant: it would demonstrate that a 7B open model can surpass much larger open and proprietary systems on video-based spatial reasoning through targeted 3D-grounded instruction data, with direct implications for embodied AI and robotics. The released dataset, models, training logs, scaling curves, and structured-reasoning examples are genuinely useful resources and appear reproducible in principle. However, the paper's own Table 1 contradicts the per-task state-of-the-art claim, and the absence of any operational description of the VSI-Bench overlap filtering leaves the central generalization claim unverified. With corrected claims and documented filtering, this could become a solid dataset-and-model study; as written, the central claim is not supported.","major_comments":[{"comment":"The claim that 'strict filtering' removes all videos overlapping with VSI-Bench is not operationalized anywhere in the paper. The authors should provide the exact procedure (e.g., scan-ID matching, hash-based deduplication, official split membership), the number of candidate videos considered and excluded, and a statement of whether any VSI-Bench video IDs appear in the released ViCA-322K. Since VSI-Bench and ViCA-322K are drawn from the same ARKitScenes/ScanNet/ScanNet++ sources, a scene-level leak would directly explain the large Absolute Distance gain (57.0 vs 30.9) without implying improved spatial cognition; this is the load-bearing uncertainty for the paper's central generalization claim.","section":"§3.1 and §4.1.2"},{"comment":"The statement 'new state-of-the-art on all eight VSI-Bench tasks' is contradicted by Table 1, where Gemini-1.5 Pro obtains 46.3 on Relative Direction and 36.0 on Route Planning versus ViCA-7B's 42.6 and 34.5. Section 4.2 acknowledges this, but the margins reported for these two tasks ('+0.2' and '+0.5') are computed against the best 7B/8B open-source model, not against the true best system, which is misleading in a state-of-the-art claim. Please revise the abstract and contributions to state 'best average score and best on six of eight tasks' (or equivalent), and report per-task margins against the actual best-performing model.","section":"Abstract, §1, §4.2, Table 1"},{"comment":"The transfer interpretation for the two held-out tasks (Relative Direction and Route Planning) is not supported by the reported numbers. The gains over the best comparable open model are +0.2 and +0.5 percentage points, which are within the run-to-run variation shown in Figure 4 (annotated differences range from 0.03 to 2.65), and no standard errors or multiple-seed results are provided for the main ViCA-7B evaluation. Please either provide repeated runs with confidence intervals and a significance test, or explicitly characterize these as near-zero transfer effects rather than evidence of learned generalization; the shared answer formats with VSI-Bench make format-calibration effects a plausible alternative explanation.","section":"§3.2 and §4.2"}],"minor_comments":[{"comment":"The gray shading is applied to ViCA-7B across all columns, but the per-column best for Relative Direction and Route Planning is Gemini-1.5 Pro; please shade per column or reword the caption so it does not imply ViCA-7B is best on every metric.","section":"Table 1 caption"},{"comment":"The text contains a recurring 'LLaV A' artifact (e.g., Table 1 and the references) that should be corrected to 'LLaVA'.","section":"Throughout"},{"comment":"The cross-reference to 'Section 5.5' is invalid; the intended target appears to be Section 5.2 or Appendix C.","section":"Appendix B, Table 3 caption"},{"comment":"The abbreviation 'MRA' (matching rate) is used without definition; please define it and state the threshold used to select numeric answers.","section":"§5.1"},{"comment":"The text says 'we process all 5,130 valid examples' while Table 2 reports valid-output counts around 4,400-4,600; clarify whether the difference corresponds to examples failing format extraction and why the numbers do not match.","section":"Appendix B"},{"comment":"Please specify how many frames are used as visual input for the Base Data and Complex Spatial Reasoning training samples, since the generation conditions (metadata-only vs 16 frames) differ from the 64-frame evaluation setting and this affects reproducibility.","section":"§3.3 and §4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is explicitly labeled a technical report and work in progress. The main issue is that the abstract's all-eight state-of-the-art claim is falsified by the paper's own Table 1, and the VSI-Bench overlap filtering is unstated. I do not see circularity in the sense of fitting answers to the benchmark, since the benchmark is external and the evaluations appear honestly reported, but the shared-scene contamination risk is real and must be resolved with data documentation rather than assertion. If the authors correct the claims and provide the filtering details, this could become a useful dataset contribution; the errors are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: the dataset is the real contribution, the average gain looks real, but the paper overstates itself and leaves the most important verification step underspecified.\n\nWhat is new: ViCA-322K is a large, 3D-metadata-grounded video QA resource built from ARKitScenes, ScanNet, and ScanNet++, plus a smaller reasoning-chain dataset. That is genuinely useful infrastructure for spatial VLM and embodied-AI work. The authors release models, code, and training logs, which is more than many papers do. The probing experiments are also honest: time instructions and 128-frame inputs do not help, and the thinking model costs 22 points on VSI-Bench. Reporting those null results is a plus.\n\nThe central number, an average VSI-Bench score of 60.6 from a 7B model, is supported by the table and is the strongest claim in the paper. The large margins on numerical tasks are consistent with training directly on the same kinds of 3D-grounded questions the benchmark asks.\n\nWhere it falls down. First, the abstract and conclusion say \"state-of-the-art on all eight VSI-Bench tasks,\" and Table 1 itself contradicts that: Gemini-1.5 Pro scores 46.3 vs 42.6 on Relative Direction and 36.0 vs 34.5 on Route Planning. The authors hedge in Section 4.2, but the headline claim is false as written and should be corrected in any revision. Second, the decontamination claim in Section 3.1 is too thin. VSI-Bench is built from the same three datasets, and the paper says it uses training splits and \"strict filtering\" with no procedure, no video-ID list, no deduplication method. Without that, the +26.1 on Absolute Distance could reflect scene or answer-distribution memorization rather than learned spatial reasoning. I do not think this is evidence of fraud; it is an unverified premise. The authors need to show the overlap check, not assert it. Third, the scaling curves in Figure 4 appear to be single runs with no seed variance, so the plateau conclusion is suggestive, not robust.\n\nThe overclaim and the missing decontamination details are the load-bearing issues. The first is easy to fix. The second requires real work but is checkable if the filtering code and video IDs are released.\n\nWho this is for: anyone building spatial video QA data or fine-tuning VLMs for indoor scene reasoning. The dataset will likely be cited regardless of the paper's final fate. It deserves a serious referee; I would send it to review with a clear request for major revision, not desk reject.","headline":"A genuinely useful 322K spatial-QA dataset and a credible average VSI-Bench gain, but the 'all eight tasks' claim is contradicted by their own Table 1 and the VSI-Bench decontamination procedure is asserted, not documented.","tokens_in":30243,"tokens_out":3907,"would_cite":false,"duration_ms":38236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B model trained on 322K spatial QA pairs outperforms much larger models on all eight VSI-Bench tasks.","keywords":["video spatial reasoning","VSI-Bench","vision-language models","3D metadata supervision","instruction tuning","spatial QA dataset","interpretable reasoning","indoor scene understanding"],"falsifier":"Check every VSI-Bench test video against the sourcing logs of ARKitScenes, ScanNet, and ScanNet++ used to build ViCA-322K; if any test scene appears in training, or if the model scores much above chance on rephrased questions and shuffled answer templates from the same scenes, the generalization claim is weakened. The cleanest version is to retrain on the same 322K data after removing all videos from scenes that appear in VSI-Bench and see whether the 26.1-point absolute-distance margin survives.","tokens_in":29248,"feed_emoji":"📐","tokens_out":7665,"duration_ms":70613,"temperature":0.7,"pith_summary":"Video-based spatial reasoning—judging distances, sizes, order, and layout from real indoor video—remains hard for vision-language models. The paper addresses this with ViCA-322K, a dataset of 322,003 question-answer pairs built from 3D-annotated indoor videos, and ViCA-7B, a 7B model fine-tuned on it. The authors claim ViCA-7B reaches the highest average score on all eight VSI-Bench tasks, topping larger open models and proprietary APIs, with the largest gain on absolute distance. They also show that adding complex reasoning data helps more than scaling base question count, and that forcing the model to explain its reasoning makes its process visible at a measurable accuracy cost. If the claim holds, targeted spatial supervision is a more effective route to embodied spatial competence than simply scaling model size or input frames.","feed_headline":"7B video model tops all eight spatial-reasoning tests","feed_subtitle":"322K geometry-grounded QA pairs lift average VSI-Bench score to 60.6, beating 72B and proprietary models.","key_machinery":"The load-bearing object is the ViCA-322K training mixture: 281,359 base question-answer pairs whose answers can be computed exactly from 3D bounding-box metadata (object count, relative and absolute distance, object size, appearance order, room size) plus 40,644 complex-reasoning pairs generated from video frames and metadata, covering conversations, furniture placement, occupancy, wheelchair access, and spatial description. The design deliberately omits training questions for relative direction and route planning, so gains on those VSI-Bench tasks serve as evidence of transfer. The fine-tuning base is an existing 7B video-language model, and the comparison benchmark is VSI-Bench's eight tasks.","core_discovery":"The paper's central claim is that a 7B vision-language model fine-tuned on a purpose-built spatial QA dataset can set a new top performance on VSI-Bench's eight video-based spatial tasks, outperforming not only similar-size open models but also 72B-scale models and proprietary systems. On the absolute-distance task it reports 57.0 versus 30.9 for the next best model, a 26.1-point margin, and its average is 60.6. The authors attribute the numerical-task gains to supervision generated directly from 3D oriented bounding boxes, and the appearance-order gain to temporal first-appearance detection used during data preparation. They further report that excluding complex reasoning data lowers the average to 55.4, and that a variant trained to articulate explicit thoughts drops to 38.5 while producing readable step-by-step reasoning.","pith_inferences":["If the transfer result is real, the six-task base could be pruned further: holding out other task families and measuring transfer would reveal which spatial skills are genuinely shared.","The failure of extra frames and timestamps may indicate a fixed visual-token bottleneck rather than a lack of temporal information; a testable next step is variable token allocation or temporal pooling before the language model.","Because the complex-reasoning data is generated by an LLM from metadata or frames, its benefit could come from linguistic structure rather than new visual perception; training on the same CSR text with scrambled frame order would separate these.","The paper's own limitation section restricts generalization to indoor scenes and notes the LLM-generated data may carry biases, so a natural extension is to test the same training recipe on outdoor or egocentric robot video."],"forward_implications":["A 7B open model can surpass much larger and proprietary models on a video spatial-reasoning benchmark after targeted fine-tuning, so parameter count is not the limiting factor for this capability.","Six supervised spatial tasks transfer to two held-out tasks, relative direction and route planning, suggesting the learned representation is spatial rather than task-specific.","Complex Spatial Reasoning data adds about five average points over base-only data and widens the gap as data grows, so language-grounded structural supervision complements exact geometric supervision.","Adding timestamps or doubling input frames does not improve scores, so current architectures are not exploiting extra temporal information as provided.","Explicit reasoning supervision makes model decisions inspectable but costs 22.1 average points, concentrated in numerical tasks, implying an accuracy-interpretability tradeoff in single-stream decoding."],"supporting_citations":[{"why":"Defines VSI-Bench, the eight-task benchmark every model is compared against.","marker":"Yang et al. 2024"},{"why":"Supplies ARKitScenes, one of the three 3D-annotated indoor video sources.","marker":"Baruch et al. 2021"},{"why":"Supplies ScanNet, the second 3D-annotated indoor video source.","marker":"Dai et al. 2017"},{"why":"Supplies ScanNet++, the third indoor video source with high-fidelity 3D scans.","marker":"Yeshwanth et al. 2023"},{"why":"Provides LLaVA-Video-7B-Qwen2, the base vision-language model that is fine-tuned into ViCA-7B.","marker":"Zhang et al. 2024b"},{"why":"The GPT-4o-mini model that generates the complex-reasoning question-answer pairs.","marker":"OpenAI 2024"},{"why":"The YOLO detector used to establish precise first-appearance times for appearance-order questions.","marker":"Redmon et al. 2016"},{"why":"Generates the candidate step-by-step reasoning chains from which the ViCA-Thinking-2.68K dataset is built.","marker":"Gemini 2.5 Pro Preview 03-25"}],"fun_headline_variants":["7B model tops all eight spatial tasks, beating 72B rivals","322K spatial QA pairs teach a 7B model to beat 72B on video tests","7B spatial model outperforms 72B on every VSI-Bench task","Spatial-reasoning dataset of 322K QAs gives 7B model edge over 72B","7B ViCA model sets new record on all eight spatial video tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that VSI-Bench measures generalization rather than memorization: the paper asserts strict filtering to exclude benchmark videos but never specifies the filtering procedure, and if any benchmark scenes or answer-format cues leaked into the 322K training data, the reported margins could partly reflect overlap.","fun_headline_variants_meta":{"raw":{"variants":["7B model tops all eight spatial tasks, beating 72B rivals","322K spatial QA pairs teach a 7B model to beat 72B on video tests","7B spatial model outperforms 72B on every VSI-Bench task","Spatial-reasoning dataset of 322K QAs gives 7B model edge over 72B","7B ViCA model sets new record on all eight spatial video tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002086,"raw_usage":{"total_tokens":8094,"prompt_tokens":912,"completion_tokens":7182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":7070}},"tokens_in":528,"tokens_out":7182,"duration_ms":49293,"temperature":1.0,"reasoning_tokens":7070,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:20.446757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check every VSI-Bench test video against the sourcing logs of ARKitScenes, ScanNet, and ScanNet++ used to build ViCA-322K; if any test scene appears in training, or if the model scores much above chance on rephrased questions and shuffled answer templates from the same scenes, the generalization claim is weakened. The cleanest version is to retrain on the same 322K data after removing all videos from scenes that appear in VSI-Bench and see whether the 26.1-point absolute-distance margin survives.","supporting_citations":[],"review_version":1}