{"id":"73caa4bb-f14b-47cf-9ae0-6d59b5214651","arxiv_id":"2505.00788","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SpatialLLM, trained with 3D-aware orientation and distance data across multiple stages, scores 62.7% on the new SpatialVQA benchmark, surpassing GPT-4o by 8.7%.","lead":"This paper presents SpatialLLM, a large multimodal model trained with new 3D-informed data that includes object orientation, distance, and spatial relationships, plus a new benchmark called SpatialVQA for evaluating 3D spatial reasoning. The authors report that SpatialLLM beats GPT-4o by 8.7% on this benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SpatialVQA's rule-based questions include several types solvable from 2D cues; the 8.7% SOTA margin may not measure 3D reasoning until a 2D-cue oracle is ruled out.","rationale":"The reader's weakest assumption identifies exactly the load-bearing risk: SpatialVQA's rule-based labels may be answerable from 2D cues, making the benchmark an unreliable measure of 3D spatial reasoning. My analysis of the seven question types in Appendix B strengthens this concern. 'Closer to camera' is a depth-ordering task solvable by monocular scale and occlusion; 'facing camera' is the visible side of an object; 'higher' and 'on which side' are often recoverable from 2D geometry; 'same direction' can be approximated from projected orientation. Thus the paper's Sec. 3.2.2 assertion that all questions require 3D awareness is unsupported, and no 2D-only baseline is reported. This is more fundamental than the reader's secondary points (error bars, external benchmarks, GPT-4o prompt verification) because if the benchmark is 2D-solvable, the headline result is uninterpretable regardless of those fixes. I do not see a separate internal inconsistency in the training recipe; the ablations are informative and the roadmap is plausible. The proposed concrete test is cheap and decisive: a hand-coded 2D oracle using the same images and 2D boxes would establish an upper bound for non-3D solvability. If that oracle fails to reach high accuracy, the benchmark likely does require 3D reasoning, and the paper's central claim becomes credible. Because the concern is currently unresolved and the reader's conditional verdict already embeds this condition, I recommend no change to the verdict.","tokens_in":17823,"tokens_out":5939,"duration_ms":63100,"concrete_test":"Build a 2D-cue oracle on SpatialVQA using Omni3D 2D bounding boxes (or a detector) and simple rules: predict 'closer to camera' by larger 2D box area; 'higher' by lower image y-coordinate; 'on which side' by relative image-plane position; 'facing camera' by the visible side of the object from category-specific appearance; 'same direction' by comparing 2D box aspect ratios and orientation; leave 'closer to object' and 'facing object' to a 2D depth heuristic (e.g., relative box size and vertical position). Report per-type and average accuracy against the 3D-derived ground truth. If the oracle exceeds roughly 55-60% average, or exceeds 80% on any non-'closer to object' category, the claim that SpatialVQA cannot be answered from 2D cues is falsified, and the 62.7% headline cannot be attributed to 3D spatial reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SpatialLLM surpasses GPT-4o by 8.7% and SpatialVLM by 10.5% on 3D spatial reasoning rests entirely on SpatialVQA. That benchmark is generated by rule-based code from Omni3D 3D bounding boxes (Sec. 3.2.2, Appendix B), and the paper asserts that all questions 'cannot be answered from 2D spatial reasoning only' (Sec. 3.2.2). This assertion is not supported by the listed question types: 'Closer to camera' can often be answered from apparent size and occlusion; 'Facing camera' reduces to recognizing which side of an object is visible in the 2D image; 'Higher' and 'On which side' are largely readable from image-plane coordinates; and 'Same direction' can be approximated from projected 2D orientation. Only 'Closer to object' and possibly 'Facing object' genuinely require 3D relationships between objects. If a substantial fraction of SpatialVQA is 2D-solvable, the benchmark measures 2D spatial perception and template recognition rather than the claimed 3D spatial reasoning. In particular, the +10.7% gain from 3DI-Ft1M instruction tuning could reflect memorizing question-answer templates that resemble the benchmark, and the headline comparison to GPT-4o loses its meaning. The paper provides no control baseline that uses only 2D cues, so the benchmark's construct validity is the most load-bearing unresolved issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpatialLLM, a large multimodal model designed for 3D spatial reasoning, and introduces SpatialVQA, a 1,323-question benchmark generated from Omni3D 3D bounding boxes. The method combines a compound design: 3D-informed probing data for alignment, 3D-informed instruction-tuning data, a mixed vision encoder (CLIP+DINOv2), and an upgraded LLM (Llama3-8B). The paper reports 62.7% accuracy on SpatialVQA, surpassing GPT-4o by 8.7% and SpatialVLM by 10.5%, and attributes the gains to architecture improvements (1.3%) and 3D-informed data (13.7%). The systematic ablation over training stages and data types is the paper's main methodological contribution.","tokens_in":18029,"tokens_out":6126,"duration_ms":52794,"significance":"The paper's strengths are its systematic coverage of the design space (data type, training stage, vision encoder, LLM) and the construction of large 3D-informed training datasets (3DI-Pb-OI1M, 3DI-Pb-IN166K, 3DI-Ft1M). The idea of injecting 3D orientation probing data at the alignment stage and the 13.7% gain from data-side changes are, if confirmed, nontrivial and useful for practitioners. The new SpatialVQA benchmark, however, is the sole basis for all headline comparisons; because its construct validity is not demonstrated and the evaluation is on the same family of rule-generated data used for training, the central claim of state-of-the-art 3D spatial reasoning is not yet established. With external validation and better statistical controls, the contribution would be a valuable reference for the community.","major_comments":[{"comment":"The central assertion that all SpatialVQA questions 'cannot be answered from 2D spatial reasoning only' is not supported by the question inventory. 'Closer to camera' is largely determined by apparent size and occlusion; 'Facing camera' reduces to which side is visible in the 2D image; 'Higher' and 'On which side' are mostly readable from image-plane coordinates; 'Same direction' can be approximated from projected 2D orientation. Only 'Closer to object' and possibly 'Facing object' seem to require true 3D relationships. The paper provides no 2D-cue oracle baseline, so the construct validity of the benchmark is unestablished. I request a control evaluation: implement a rule-based oracle using only 2D signals (2D box sizes, image-plane centers, occlusion/ordering, projected orientation) and report its accuracy per question type, along with the fraction of questions that are ambiguous or degenerate under 2D cues. This is necessary to interpret the headline 62.7% and the 8.7% margin over GPT-4o in Table 1.","section":"Sec. 3.2.2 / Appendix B"},{"comment":"The training data (3DI-Pb, 3DI-Ft1M) and the evaluation benchmark (SpatialVQA) are generated by the same group using the same rule-based conversion of 3D bounding box annotations (Omni3D for the benchmark; OpenImages/ImageNet3D for training) and the same question templates. The +10.7% from 3DI-Ft1M instruction tuning (Fig. 6) may therefore reflect learning benchmark-specific templates and label distributions rather than general 3D spatial reasoning. The manuscript does not report whether SpatialVQA images overlap with training images, nor does it evaluate on any independent benchmark. Please add (i) a check for image/annotation overlap between SpatialVQA and all training sets; (ii) evaluation on at least one external spatial reasoning benchmark or a human-annotated subset; and (iii) a discussion of how much of the gain persists when the evaluation template is varied.","section":"Sec. 3.3.2 / Table 2 / Appendix A"},{"comment":"All conclusions are based on single-run accuracies on a 1,323-question set. For the overall accuracy, the approximate 95% binomial confidence interval is ±2.7 percentage points; for the per-type subsets (roughly 240 questions each) it is about ±6.3 points. Thus the reported differences of 0.3% (CLIP+DINOv2 vs CLIP+MAE) and 1.0% (LLaVA-v1.5 to Llama3) are within sampling noise, yet the roadmap in Fig. 6 treats them as design recommendations. Report multiple seeds or bootstrap confidence intervals, and correct for the multiple comparisons inherent in selecting a model on the same benchmark that produced the headline number.","section":"Table 2 / Fig. 6"},{"comment":"The comparison with GPT-4o is not quantified transparently. In Fig. 7, GPT-4o's response is a refusal ('without the ability to visually assess the image'), which is a limitation of the interaction protocol, not necessarily a spatial reasoning error. If refusals are scored as incorrect, the 8.7% margin over GPT-4o is inflated. The paper must report the exact prompt, the number of calls, retries, refusal detection, and the accuracy after excluding refusals or providing the image again. The same protocol should be documented for Claude 3.5 Sonnet.","section":"Fig. 7 / Table 1"}],"minor_comments":[{"comment":"Typo: 'for for complex spatial relationships' should read 'for complex spatial relationships'.","section":"Sec. 1"},{"comment":"'3DI-Pb-1N166K' appears to be a typo for '3DI-Pb-IN166K'.","section":"Fig. 6"},{"comment":"'azimuth elevator' should be 'azimuth elevation'.","section":"Fig. 5(d)"},{"comment":"SpatialRGPT is cited as [14,16]; reference [14] is SpatialVLM, which is a different method. Please correct the citation.","section":"Related Work"},{"comment":"The angle threshold used to define 'same direction' and 'on which side' is not specified; please state the exact rule for completeness and reproducibility.","section":"Appendix B"},{"comment":"The statement that 'We pretrain the visual connector only while freezing other modules' is inconsistent with the Stage 0 rows of Table 2, which update LoRA layers of the CLIP encoder; please clarify which modules are updated at each stage.","section":"Supplementary Sec. A"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is authored by the same team and generated with the same rule-based templates as the training data; I would want to see independent validation before interpreting the SOTA claim as evidence of general 3D reasoning. Also, the claim of being 'the first to curate VQA data that incorporate 3D orientation relationships on real images' should be checked against prior work on 3D pose-aware VQA, including the authors' own NeurIPS 2024 paper on 3D-aware VQA (reference [60]), which uses 3D pose annotations. This does not change my recommendation, but it is worth verifying with the authors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious engineering paper with a clear message—3D-informed data at alignment and instruction tuning helps—but the headline numbers rest on a benchmark that may be partly solvable from 2D cues, and the same benchmark was used to pick the design. The SOTA claim is not yet supported.\n\nWhat's actually new: the first VQA set targeting 3D orientation relationships on real images, and a systematic compound design search that separates architecture, data, and training-stage effects. The roadmap is genuinely informative: 3D-informed instruction tuning gives +10.7%, human-annotated ImageNet3D alignment another +3%, and architecture changes only +1.3%. Those ablations are the real contribution, along with the benchmark itself. Credit where due: this is reproducible groundwork for a line of work.\n\nSoft spots, in proportion. The main one is the benchmark. SpatialVQA is generated by rule-based code from Omni3D boxes, and the paper asserts that all questions \"cannot be answered from 2D spatial reasoning only\" without demonstrating it. The stress-test concern is fair: 'closer to camera' is often answerable from apparent size and occlusion, 'higher' and 'on which side' are largely image-plane heuristics, and even 'same direction' can be approximated from 2D orientation. Only a fraction of the question types genuinely force 3D reasoning. No 2D-cue control baseline is provided, so the 8.7% margin over GPT-4o might be partly template recognition and 2D heuristics. That's load-bearing, not minor.\n\nSecond, the same benchmark is used for design selection and final reporting, with no error bars. That makes the reported gains look fragile. Third, the GPT-4o qualitative example includes a refusal that suggests an incomplete prompt, which weakens the comparison. Fourth, the SpatialVLM baseline is a third-party implementation, so the +10.5% number is not apples-to-apples with the original model. Finally, no external spatial benchmark (GSR-Bench, ScanQA) is used, so the claim of generalizable 3D reasoning is untested.\n\nThese are all addressable. The recipe claim likely survives; the SOTA framing does not. This paper deserves a serious referee: it ships a new benchmark, a clear ablation study, and a reproducible training pipeline, but the reviewer should push for a 2D-cue oracle, confidence intervals, external benchmarks, a proper GPT-4o prompt, and release of code, data, and weights. I'd engage with it, and I'd cite it once the benchmark is cleaned up.","headline":"A useful training recipe and a new orientation-heavy benchmark, but the benchmark's construct validity is not established, so the headline SOTA claim overreaches.","tokens_in":18664,"tokens_out":1678,"would_cite":true,"duration_ms":18204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpatialLLM claims a staged 3D-data recipe lifts image-only spatial reasoning to 62.7%, beating GPT-4o by 8.7 points.","keywords":["large multimodal models","3D spatial reasoning","spatial visual question answering","3D object orientation","3D-informed data","instruction tuning","multimodal alignment","SpatialVQA"],"falsifier":"A pure 2D heuristic that answers SpatialVQA's distance and side questions from bounding-box area, pixel position, and occlusion would settle the claim: if it reaches or exceeds 62.7%, the benchmark does not isolate 3D reasoning.","tokens_in":17541,"feed_emoji":"🧭","tokens_out":10226,"duration_ms":88555,"temperature":0.7,"pith_summary":"The paper tries to establish that the missing ingredient in large multimodal models' 3D spatial reasoning is not larger models or specialized 3D input modules, but a deliberate staging of 3D-informed text data across ordinary training stages. To show this it introduces SpatialLLM, trained from images alone with two new data types: object-level 3D probing questions (depth, distance, azimuth/elevation) and one million multi-turn conversations about 3D spatial relationships. On the paper's new SpatialVQA benchmark, the recipe scores 62.7% average accuracy, 8.7 points above GPT-4o and 10.5 points above the spatial-centric SpatialVLM. Controlled ablations attribute 13.7 points of the gain to data placement and only 1.3 points to architecture, with human-annotated orientation data at the alignment stage working better than pseudo-labels. If correct, this gives the community a reproducible roadmap for adding 3D awareness to LMMs without extra depth maps, masks, or boxes at inference.","feed_headline":"SpatialLLM beats GPT-4o on 3D spatial questions by 8.7 points","feed_subtitle":"Adding 3D probing data at alignment and spatial conversations at tuning reaches 62.7% on SpatialVQA.","key_machinery":"The load-bearing mechanism is the compound training recipe, not any single network module. Concretely: (1) 3DI-Pb-IN166K, a set of object-level 3D probing questions converted from human-annotated ImageNet3D poses, is inserted at the multimodal alignment stage; (2) 3DI-Ft1M, one million instruction-tuning conversations about distances, facing directions, and relative 3D positions, is inserted at instruction tuning; (3) the architecture is a LLaVA-style model with a CLIP+DINOv2 mixed vision encoder and Llama3-8B, with the encoder kept frozen; and (4) SpatialVQA, 1,323 rule-generated questions built from Omni3D 3D bounding boxes, supplies the yardstick for distance, orientation, and combined spatial reasoning. The design-space search shows that moving either dataset to the wrong stage, or replacing human pose annotations with pseudo-labels, lowers accuracy.","core_discovery":"The paper's central claim is that 3D spatial reasoning in an LMM is primarily a data-placement problem. Starting from LLaVA-v1.5 at 47.7% on SpatialVQA, the paper moves through a roadmap: mixing a self-supervised DINOv2 encoder's features with CLIP and upgrading the LLM to Llama3-8B adds about 1.3 points; adding 3D-informed instruction conversations (3DI-Ft1M) at the tuning stage adds 10.7 points; and adding human-annotated 3D orientation probing data (3DI-Pb-IN166K) at multimodal alignment adds another 3 points, reaching 62.7%. The authors report that this is the first VQA data to include 3D orientation relationships on real images, and they show that pre-training the vision encoder on 3D probing data (stage 0) does not help, so the final design keeps visual features frozen.","pith_inferences":["Editorial inference: the 86.3% distance accuracy versus 52.9% orientation accuracy suggests the field's next bottleneck is orientation perception; datasets with higher-quality 6D pose labels, not more conversations, are likely to drive the next jump.","Editorial inference: since stage-0 vision encoder tuning hurt performance, injecting 3D awareness through adapters or late fusion may be safer than fine-tuning the encoder, and the paper's negative result points there.","Editorial inference: if the 13.7-point dominance of data over architecture reproduces on larger backbones, training-stage data placement should become a standard reporting axis for multimodal model papers, not just spatial ones."],"forward_implications":["Adding 3D-informed instruction tuning data alone yields a 10.7-point jump on SpatialVQA, making it the single strongest design decision.","Adding human-annotated 3D pose probing data during multimodal alignment adds 3 more points and outperforms semi-automated pseudo-labels.","Architecture upgrades contribute about 1.3 points total, so future gains are more likely to come from data composition than from new encoders or LLMs.","Fine-tuning the vision encoder on 3D data before alignment is counterproductive; frozen pretrained encoders remain the better design.","The final image-only model reaches 62.7% average accuracy and 86.3% on 3D distance questions, while orientation reasoning lags at 52.9%."],"supporting_citations":[{"why":"The proprietary model that SpatialLLM is compared against and outperforms on SpatialVQA.","marker":"[1]"},{"why":"Supplies the Omni3D images and 3D bounding-box annotations from which SpatialVQA is generated.","marker":"[11]"},{"why":"The spatial-centric baseline whose 3D-informed instruction tuning lacks orientation data and is outperformed.","marker":"[14]"},{"why":"Prior spatial-reasoning model whose rule-based VQA generation and extra-input approach SpatialLLM builds on and does not require.","marker":"[16]"},{"why":"Provides the LLaVA-v1.5 architecture and standard CC558K/Mix665K data recipe used as the starting point.","marker":"[39]"},{"why":"Provides human-annotated ImageNet3D pose data and the pretrained 6D pose estimator used to build 3D orientation data.","marker":"[45]"},{"why":"Self-supervised vision encoder whose features, mixed with CLIP, give a small spatial-reasoning gain.","marker":"[47]"},{"why":"CLIP vision encoder that forms the base of the mixed encoder and the standard LMM visual backbone.","marker":"[48]"}],"fun_headline_variants":["SpatialLLM: 3D spatial reasoning tops GPT-4o by 8.7%","First 3D-orientation VQA on real images: SpatialLLM outperforms GPT-4o","From 47.7% to 62.7%: SpatialLLM's 3D data recipe beats GPT-4o","SpatialLLM proves 3D data placement is key: 8.7% edge over GPT-4o","SpatialLLM: New 3D VQA data lifts LMM reasoning past GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers rest on SpatialVQA's rule-generated answers being correct and genuinely 3D; if those answers are noisy or reachable from 2D cues, the gap to GPT-4o measures something other than 3D reasoning.","fun_headline_variants_meta":{"raw":{"variants":["SpatialLLM: 3D spatial reasoning tops GPT-4o by 8.7%","First 3D-orientation VQA on real images: SpatialLLM outperforms GPT-4o","From 47.7% to 62.7%: SpatialLLM's 3D data recipe beats GPT-4o","SpatialLLM proves 3D data placement is key: 8.7% edge over GPT-4o","SpatialLLM: New 3D VQA data lifts LMM reasoning past GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3136,"prompt_tokens":1031,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":1961}},"tokens_in":647,"tokens_out":2105,"duration_ms":15194,"temperature":1.0,"reasoning_tokens":1961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:35:25.620493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A pure 2D heuristic that answers SpatialVQA's distance and side questions from bounding-box area, pixel position, and occlusion would settle the claim: if it reaches or exceeds 62.7%, the benchmark does not isolate 3D reasoning.","supporting_citations":[{"cited_title":"Omni3D: A large benchmark and model for 3D object detection in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the Omni3D images and 3D bounding-box annotations from which SpatialVQA is generated."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-v1.5 architecture and standard CC558K/Mix665K data recipe used as the starting point."},{"cited_title":"Imagenet3d: Towards general-purpose object-level 3d understanding","cited_arxiv_id":null,"evidence_quote":"Provides human-annotated ImageNet3D pose data and the pretrained 6D pose estimator used to build 3D orientation data."},{"cited_title":"Dinov2: Learning robust visual features without supervision","cited_arxiv_id":null,"evidence_quote":"Self-supervised vision encoder whose features, mixed with CLIP, give a small spatial-reasoning gain."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP vision encoder that forms the base of the mixed encoder and the standard LMM visual backbone."}],"review_version":1}