{"id":"7d4a80be-ab45-4fc4-aa7e-60511b6f01c6","arxiv_id":"2511.19119","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MonoSR is a 1M-question benchmark for spatial reasoning from single photos across indoor, outdoor, and object-centric scenes; current VLMs score roughly 30-40%, and giving models 3D box coordinates lifts them near perfect.","lead":"This paper introduces MonoSR, a dataset of over one million spatial-reasoning questions about single photos of indoor, outdoor, and object-centric scenes. It also benchmarks current vision-language models and tests how much extra 2D and 3D information helps them answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MonoSR's ground-truth answers inherit Omni3D 3D-box labels without independent validation; if those labels are noisy, benchmark scores and auxiliary-info conclusions are compromised.","rationale":"I read MonoSR as a dataset-construction paper whose central claim is that the 1M+ QA pairs are reliable ground truth for monocular spatial reasoning, and that experiments on them reveal VLM limitations and the role of explicit geometry. The most load-bearing condition is therefore the correctness of the Omni3D 3D bounding boxes from which every answer is derived. The paper asserts deterministic derivation and describes human validation, but it provides no independent check of the upstream labels. Since Omni3D aggregates many datasets with different sensors and label conventions, the risk is real and concrete: even a small fraction of bad boxes can systematically corrupt numeric questions and borderline categorical questions. The auxiliary-information experiment compounds this because the same boxes serve both as answer source and as the experimental '3D bbox' condition; the near-perfect numbers there are partly a circularity, though the paper acknowledges that ground-truth spatial information is impractical. I do not see this as fatal: the benchmark can be salvaged by per-source validation, released provenance, and error bars. This is exactly the reader's weakest assumption, so my stress-test does not move the verdict. I considered whether the auxiliary-information leakage should be the primary concern, but the paper openly discusses ground-truth dependence, whereas the unverified Omni3D label validity is more fundamental and less acknowledged.","tokens_in":21683,"tokens_out":5266,"duration_ms":60316,"concrete_test":"Stratified audit: take ~1,000 MonoSR QA pairs per Omni3D source dataset (KITTI, nuScenes, SUN RGB-D, ARKitScenes, Hypersim). Re-derive 3D boxes from source-level raw annotations/depth (e.g., re-fit oriented boxes to SUN RGB-D/ARKitScenes depth maps; recover KITTI/nuScenes official labels into the same camera frame used by MonoSR). Regenerate answers using MonoSR's exact pipeline and compute answer-flip rates for numeric and categorical tasks. Also report human annotator agreement with the original answers when given only the image. If flip rate exceeds ~5% on numeric tasks or ~2% on categorical tasks, the paper should report per-source error bars, release a filtered subset, and soften the auxiliary-information conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MonoSR's central artifact is a 1M+ QA dataset whose answers are claimed to be ground truth because they are deterministic functions of Omni3D 3D bounding boxes (Sec. 3.2). This is an inherited-label validity claim, and the paper does not independently verify the boxes: the reported 'geometric consistency checks against 3D annotations' only verify that the pipeline computed what it was given, and the human validation is described as 'large, randomly sampled' with no N, protocol, or error rate. Omni3D aggregates heterogeneous sources (KITTI, nuScenes, SUN RGB-D, ARKitScenes, Hypersim) with different sensors, depth quality, label protocols, coordinate frames, and annotation noise; distant/occluded objects are exactly where such labels degrade. Any bias or noise in center, size, or orientation propagates directly into numeric answers (Dist/Size), relative labels (SR/PR), and occlusion judgments. The auxiliary-information experiment then uses those same boxes as the 3D-bbox condition, so its near-ceiling numbers (Tab. 2, e.g., 0.986 object-centric) are partly circular: the model is handed the answer source. This does not make the paper internally inconsistent, but it makes the benchmark's validity and the headline conclusions contingent on a label-quality assumption that is currently unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MonoSR, a large-scale monocular spatial reasoning dataset constructed from Omni3D: over 1.02M QA pairs from 230K+ images spanning indoor, outdoor, and object-centric domains, with 8 task types organized into three cognitive levels. Answers are derived deterministically from Omni3D 3D ground-truth bounding boxes via scene graphs and handcrafted templates, with LLM-based paraphrase for linguistic diversity. The paper benchmarks six open- and closed-source VLMs on MonoSR, reports generally low accuracies, and fine-tunes Qwen2.5-VL-3B under different auxiliary information conditions (scene type, 2D visual prompts, 3D bounding boxes), finding large gains from explicit 3D geometry. The central claim is that MonoSR provides a foundation for open-world monocular spatial reasoning and that current VLMs primarily lack fine-grained 3D geometric information rather than object recognition.","tokens_in":22006,"tokens_out":5160,"duration_ms":46086,"significance":"If the dataset is released and its labels independently validated, MonoSR would fill a real gap: most prior 3D spatial reasoning benchmarks are indoor-only, multi-view, or reconstruction-based. The deterministic answer generation from 3D boxes is a reproducibility strength, and the controlled auxiliary-information comparison is a useful protocol for guiding monocular 3D reasoning research. The VLM evaluation is broad across model families. However, the contribution is currently conditional: the dataset artifact is not shipped, inherited Omni3D label noise is unquantified, and the headline 3D-bbox conclusion is partly circular because the model is given the same ground-truth geometry used to generate the answers. These issues are load-bearing for the paper's central claims.","major_comments":[{"comment":"MonoSR's central artifact is not linked or downloadable. The paper references a project page and code links but provides no dataset URL, release license, split files, or download instructions. Without the dataset, the 1M+ QA pairs, the VLM benchmark, and the fine-tuning experiments cannot be reproduced or used by the community. A dataset paper must ship the dataset (or a hosted subset) and provide a datasheet with splits, category frequencies, and annotation protocols.","section":"Sec. 3.3 / Availability"},{"comment":"All QA answers are derived from Omni3D 3D bounding boxes, but the paper provides no independent verification of those boxes. The reported 'geometric consistency checks against 3D annotations' only verify pipeline self-consistency, not external correctness. The human validation is described only as a 'large, randomly sampled subset' with no N, sampling method, per-task error rates, or inter-annotator agreement. Because Omni3D aggregates heterogeneous datasets (KITTI, nuScenes, SUN RGB-D, ARKitScenes, Hypersim) with different sensors and label conventions, label noise propagates directly into numeric answers, relative labels, and occlusion judgments. Quantify this error and its effect on benchmark numbers.","section":"Sec. 3.2 / Sec. 3.3"},{"comment":"The 3D-bbox auxiliary condition is circular: the model receives the exact center, size, and orientation from the same ground-truth boxes used to generate the answers. Near-ceiling accuracy (e.g., 0.986 object-centric overall) is therefore expected by construction. The conclusion that 'the primary bottleneck is not object recognition, but the recovery of fine-grained spatial attributes' is too strong; the condition is an upper bound, not a realistic input signal. Add a condition using predicted 3D boxes from an off-the-shelf monocular detector, or rephrase the conclusion accordingly.","section":"Sec. 5.3 / Tab. 2"},{"comment":"Benchmark numbers are single-run with no error bars or significance tests, and the 10% numeric tolerance is arbitrary without sensitivity analysis. Some reported numbers are extremely low (e.g., High-Dist indoor 0.002 for Gemini-2.5-Pro; object-centric overall 0.069 for ChatGPT-4), which is consistent with evaluation instability. Additionally, 'deterministic decoding with temperature 1.0' is internally contradictory: temperature 1.0 is stochastic. Report the exact decoding settings and the variance across multiple runs or seeds.","section":"Sec. 5.1 / Eq. (1) / Sec. 11.1"}],"minor_comments":[{"comment":"Task count is inconsistent: Sec. 3.1 says 'nine tasks', while Fig. 1, Fig. 3, and Sec. 8.1 list eight. Align the text and figures.","section":"Sec. 3.1 / Fig. 1 / Fig. 3 / Sec. 8.1"},{"comment":"Typos: 'Futhuremore' (Sec. 1), 'Distnce' (Sec. 8.1), 'examplre' (Sec. 3.2), '3b Bbox' (Sec. 4), 'F oundational' (Sec. 3.2).","section":"Throughout"},{"comment":"The term 'open-vocabulary' is not defined; the dataset is built from 98 fixed Omni3D categories. Clarify whether 'open-vocabulary' refers to free-form captions or to an open-set evaluation protocol.","section":"Title / Sec. 3.3"},{"comment":"Eq. (1) is not numbered and the threshold rule is stated as a strict inequality; add a number and clarify handling of exact ties (|d-dhat|/d exactly 0.1).","section":"Sec. 5.1"},{"comment":"The situational template examples show answers labeled 'A: yes' for what appear to be numeric/volume questions; ensure the template and example answers are consistent in type and format.","section":"Fig. 12"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset/benchmark paper, but the dataset is not released and the central label-validity concern is unresolved. The auxiliary-information experiment is a useful protocol, but its headline conclusion is partly circular as written. For a journal venue, the authors should be asked to either ship the artifact and provide validation of inherited labels, or substantially soften the claims. The 'open-vocabulary' wording in the title should also be rechecked against the fixed 98-class ontology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing you should know: MonoSR is the first large-scale monocular open-world spatial reasoning benchmark I've seen, and the auxiliary-information study is a useful diagnostic. But it's a dataset paper without a working link to the dataset, and its validity claims rest on an unquantified inheritance of Omni3D's 3D boxes and an unverifiable 'large, randomly sampled' human check. Treat the headline numbers as provisional until those are on the table.\n\nWhat's genuinely new: existing spatial reasoning datasets are indoor and/or multi-view; MonoSR draws from Omni3D to cover indoor, outdoor, and object-centric scenes from single images. The QA derivation from ground-truth 3D boxes is deterministic and reproducible in principle. Using an LLM to paraphrase question text while freezing the answer is a sensible way to get linguistic diversity without label drift. The auxiliary-info study is well designed as a controlled comparison: scene info, 2D prompts, 3D boxes, and combinations, with a fixed backbone. The finding that explicit 3D boxes push object-centric accuracy to ~0.99 is telling, and the paper is honest in Sec. 6 that the current setup relies on ground-truth spatial info that is impractical for real deployment.\n\nThe soft spots, in order. First, the artifact isn't accessible: the project-page and code links are placeholders. For a resource paper, peer review can't proceed without the actual dataset and generation code. Second, human validation is a black box: no N, no protocol, no agreement metric. The 'geometric consistency checks' only verify internal consistency of the pipeline, not the validity of the Omni3D labels that answer correctness depends on. If those boxes are noisy for distant or occluded objects—which is exactly where Omni3D inherits heterogeneity from KITTI, nuScenes, SUN RGB-D, ARKitScenes, Hypersim—a meaningful fraction of the 1M answers are wrong, and both the benchmark and the auxiliary-info conclusions shift. Third, benchmark numbers lack error bars; a single deterministic run is fine for a sanity check, not for claiming which model is best. Fourth, the 3D-bbox condition is partly circular: it hands the model the exact center/size/rotation that determines the answers. That's still a valid diagnostic, but it means the 'bottleneck is perception' conclusion is closer to tautology than discovery.\n\nWho should read this: anyone working on spatial reasoning for VLMs, embodied AI, or monocular 3D understanding. It deserves a serious referee, but with a clear request: ship the dataset, report the validation numbers and protocol, and add variance estimates. Without those, my recommendation would be desk reject on the grounds that the central artifact doesn't exist; with them, it's a solid contribution.","headline":"A genuinely useful monocular spatial reasoning benchmark built on Omni3D, but the missing dataset release and unquantified label/human validation make the headline claims provisional.","tokens_in":22466,"tokens_out":4743,"would_cite":true,"duration_ms":45841,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new dataset with over one million monocular spatial-reasoning question-answer pairs shows that current vision-language models struggle most with 3D geometry, not object recognition.","keywords":["monocular spatial reasoning","vision-language models","3D bounding boxes","visual question answering","open-world dataset","scene graph","auxiliary information","benchmark"],"falsifier":"Take a random sample of MonoSR questions that require numeric values (distance, size, footprint, volume), re-project the corresponding Omni3D 3D boxes into the images using the published camera intrinsics, and compare the computed answers with measurements a human would make from the image (or from the original sensor data such as LiDAR or RGB-D where available). If a substantial fraction of answers differ by more than the paper's own 10% tolerance, the ground truth is contaminated and the reported model rankings and auxiliary-information effects are not trustworthy.","tokens_in":21587,"feed_emoji":"📐","tokens_out":6932,"duration_ms":66461,"temperature":0.7,"pith_summary":"This paper claims that existing spatial-reasoning benchmarks are too narrow—indoor-only, multi-view, or video-based—and that the most common real-world setting, a single monocular image, has been left without a large-scale training and evaluation resource. MonoSR fills that gap: over 1.02 million QA pairs built from 230,000+ images spanning indoor, outdoor, and object-centric scenes, with all answers computed deterministically from ground-truth 3D bounding boxes. The paper further claims that current vision-language models are far from adequate on this task, and, through a controlled auxiliary-information study, that the binding constraint is recovery of explicit 3D geometry rather than object recognition or scene semantics. If correct, MonoSR provides a foundation for training and benchmarking open-world monocular spatial reasoning, a capability needed for embodied AI and autonomous driving.","feed_headline":"1M single-image spatial queries expose VLM blind spots","feed_subtitle":"MonoSR derives every answer from true 3D boxes; tests show explicit geometry, not image content, drives accuracy.","key_machinery":"The load-bearing mechanism is MonoSR's QA-generation pipeline, which converts Omni3D ground-truth 3D bounding boxes into questions whose answers are deterministic functions of box center, dimensions, orientation, and camera parameters. A scene graph of fine-grained captioned objects feeds handcrafted templates; an LLM then paraphrases the question text under strict constraints that the computed answer remains unchanged. The second mechanism is the auxiliary-information experiment: the same Qwen2.5-VL-3B backbone is fine-tuned with scene information, 2D visual prompts, 3D bounding boxes, and their combinations as input, isolating which information source drives spatial accuracy.","core_discovery":"MonoSR is a large-scale monocular spatial reasoning dataset: more than 1.02 million QA pairs built from over 230,000 images, spanning indoor, outdoor, and object-centric scenes across 98 object categories. Questions are organized into three hierarchical levels—foundational 3D perception (spatial relationship, distance, size, dimension comparison), perspective-aware imagination (occlusion, viewpoint-relative relationship, object grounding), and situational reasoning (questions embedded in professional scenarios). All answers are derived deterministically from ground-truth 3D bounding boxes sourced from Omni3D, through a pipeline that filters scenes, builds a scene graph of captioned objects,","pith_inferences":["Beyond the paper: feeding the model noisy 3D estimates from a monocular detector at inference time, rather than oracle Omni3D boxes, would test whether the near-perfect auxiliary-information results survive real-world label noise.","Beyond the paper: MonoSR inherits Omni3D's box approximations of real object extents; recomputing answers from finer point clouds or meshes on a subset of scenes would test how sensitive model rankings are to box-level ground truth.","Beyond the paper: a stress test for the open-world claim is to fine-tune on MonoSR and evaluate on images from unseen sources (user photos, aerial views, synthetic scenes); a large drop would indicate dataset-specific heuristics rather than general spatial understanding.","Beyond the paper: numeric answers are judged with a fixed 10% relative-error tolerance; recomputing rankings at stricter tolerances (e.g., 1% or 5%) would show whether conclusions about VLM limits are threshold-dependent."],"forward_implications":["Existing VLMs are far from reliable monocular spatial reasoners: the best models score below 0.44 overall on MonoSR, with the weakest results on object-centric and quantitative questions.","Explicit 3D geometry, not image semantics, is the dominant driver of accuracy: a 3B model given 3D bounding boxes plus 2D prompts reaches near-perfect scores on many tasks.","MonoSR's training split provides a supervised resource for learning open-world monocular spatial reasoning across indoor, outdoor, and object-centric scenes.","Adding irrelevant or excessive auxiliary cues degrades performance, so simply supplying more context to a VLM is not a reliable route to spatial competence.","The hierarchical task levels (perception, imagination, situational reasoning) offer a diagnostic ladder for separating low-level geometric failures from reasoning failures in future models."],"fun_headline_variants":["1M spatial queries from single images expose VLM blind spots","Monocular spatial reasoning: 1M QA pairs reveal VLM limits","New dataset forces VLMs to reason in 3D from one image","Single-image spatial reasoning: 1M questions, 230K scenes, VLM stumbles","MonoSR: 1M monocular queries show geometry beats content for VLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The dataset's answers inherit Omni3D's 3D bounding-box annotations without independent re-verification; if those boxes are noisy, misaligned, or inconsistently defined across the many source datasets, a large fraction of the million answers are wrong, which would undercut both the benchmark numbers and the auxiliary-information conclusions.","fun_headline_variants_meta":{"raw":{"variants":["1M spatial queries from single images expose VLM blind spots","Monocular spatial reasoning: 1M QA pairs reveal VLM limits","New dataset forces VLMs to reason in 3D from one image","Single-image spatial reasoning: 1M questions, 230K scenes, VLM stumbles","MonoSR: 1M monocular queries show geometry beats content for VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2808,"prompt_tokens":685,"completion_tokens":2123,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":2035}},"tokens_in":429,"tokens_out":2123,"duration_ms":14801,"temperature":1.0,"reasoning_tokens":2035,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:33:39.023931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of MonoSR questions that require numeric values (distance, size, footprint, volume), re-project the corresponding Omni3D 3D boxes into the images using the published camera intrinsics, and compare the computed answers with measurements a human would make from the image (or from the original sensor data such as LiDAR or RGB-D where available). If a substantial fraction of answers differ by more than the paper's own 10% tolerance, the ground truth is contaminated and the reported model rankings and auxiliary-information effects are not trustworthy.","supporting_citations":[],"review_version":1}