{"id":"9233b520-cb30-41c1-aaea-59b090d3e390","arxiv_id":"2504.20648","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SpaRE builds a 3.4M-question synthetic spatial QA dataset from captioning sources and reports large gains on VSR and What's Up after fine-tuning Qwen2-VL models.","lead":"The paper generates millions of synthetic spatial-reasoning questions from detailed image captions and fine-tunes vision-language models on them. On some spatial benchmarks the tuned models improve sharply, but the evaluation may be inflated by overlap between training and test images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No image-level deduplication or overlap analysis against VSR/What's Up is reported, and the training sources include COCO-derived images; the headline What's Up A gain may reflect memorization rather than transferred spatial reasoning.","rationale":"Reader and I converge on the same weakest link. The training corpus is enormous (455k samples after filtering) and Localized Narratives alone contributes 232k; even a small overlap fraction could directly expose evaluation images during training. The absence of any overlap report in Section 3.3, combined with the magnitude of the 2B gain (48.8 points on What's Up A, from 44.6 to 93.4), makes memorization a live alternative explanation. I am not asserting the benchmark is contaminated; I am asserting the paper does not yet rule out contamination. This is a standard, testable benchmark-hygiene issue, not an internal inconsistency. The paper deserves credit for reporting gains on 3DSRBench and RealWorldQA, which are less likely to overlap with the caption sources, and for a careful QA-generation pipeline; those results suggest the approach has genuine value. However, the headline number cited in the abstract is the least secure result, and the conditional verdict with overlap analysis and data release is exactly right. No verdict change is needed.","tokens_in":18476,"tokens_out":7200,"duration_ms":71502,"concrete_test":"Compute exact image-level overlap between every SpaRE training source (Localized Narratives COCO/Flickr30k/ADE20K/Open Images splits, DOCCI, PixMo-Cap) and each evaluation set (VSR, What's Up A/B, 3DSRBench, RealWorldQA) using released image IDs, URLs, or perceptual hashes. Then re-evaluate SpaRE-2B/7B and their base models on the non-overlapping subset of VSR and What's Up A/B, with sample size matched to the overlap-free set. If the What's Up A advantage collapses on the disjoint subset while 3DSRBench gains persist, the central transfer claim must be revised downward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SpaRE's synthetic QA training transfers to spatial reasoning benchmarks, with a 48.8-point jump on What's Up A (44.6 to 93.4 for the 2B model). This claim requires that evaluation performance reflect learned spatial relations rather than image-specific memorization. Section 3.1.1 states that Localized Narratives is built from COCO, Flickr30k, ADE20K, and Open Images; VSR and What's Up A/B are evaluated on COCO-derived images. Section 3.3's quality assurance pipeline (deduplication, reference check, answer-description consistency, image-question consistency, spatial-relation verification) contains no image-level deduplication against evaluation benchmarks, and no overlap analysis is reported anywhere. Since the QA pairs are synthesized from captions that explicitly state spatial relations for a given image, any COCO image appearing in both training and evaluation lets the model exploit image-level cues. The modest gains on 3DSRBench and RealWorldQA suggest some transfer, but those benchmarks cannot rescue the headline claim if the What's Up A gain is contamination-driven. The abstract's '49% performance gain' is therefore not yet supportable without a disjoint-image evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SpaRE, a synthetic spatial-reasoning VQA dataset built by prompting Qwen2.5-3B to extract QA pairs from hyper-detailed image descriptions in Localized Narratives, DOCCI, and PixMo-Cap. The authors fine-tune Qwen2-VL-2B and Qwen2-VL-7B on 455k samples containing 3.4M QA pairs and report large improvements on spatial reasoning benchmarks, including an increase from 44.6 to 93.4 on What's Up A for the 2B model, while general benchmarks remain stable. The central claim is that caption-derived synthetic QA data transfers spatial reasoning to unseen settings without harming general VL performance.","tokens_in":18715,"tokens_out":6831,"duration_ms":58457,"significance":"If the results hold, this is a valuable contribution: it suggests that a simple caption-to-QA pipeline can address spatial-reasoning data scarcity at scale. The paper is strong in empirical scope, including multiple seeds, a range of spatial and general benchmarks, human evaluation of data quality, and ablations of the generation model. The headline improvement on COCO-derived benchmarks, however, is threatened by the lack of any overlap analysis with the training sources, which include COCO images via Localized Narratives. The paper's machine-checked experiments and reproducible recipes (open weights, open data planned) are assets, but the contamination risk must be resolved before the claim is fully supported.","major_comments":[{"comment":"Localized Narratives is built from COCO, Flickr30k, ADE20K, and Open Images images, and the VSR and What's Up benchmarks evaluate on COCO-derived images. The paper reports no overlap analysis between the 455k training images and these evaluation images, and the deduplication in Section 3.3 only removes duplicate QA pairs within a sample, not duplicate images across the train/test boundary. Because the QA pairs are synthesized from captions that state spatial relations for a given image, any COCO image appearing in both training and evaluation allows the model to exploit image-level cues. The abstract's headline claim of a 49% gain on What's Up A therefore requires either a demonstrated absence of overlap or a re-evaluation on a disjoint image set.","section":"Section 3.1.1, 4.2.1, Table 3"},{"comment":"The GPT-4o and GPT-4o-mini rows in Table 3 are computed on 100-example subsamples of VSR, What's Up A, and What's Up B, while the SpaRE models and other baselines are evaluated on the full benchmarks. The main text does not disclose this subsampling, making the comparison rows appear to be on equal footing. The paper should either run these models on the full benchmarks or clearly mark the subsample-based rows in the table and state the associated sampling error.","section":"Section 4.3, Appendix F"},{"comment":"The analysis that motivates the paper is framed as showing that spatial relations are generally rare in VQA datasets, but Table 2 reports Objects365-YorN as 94.35% spatial-related samples, and lists VSR as one of the 'VQA datasets in the supervised fine-tuning set used by InternVL2.' VSR is described in Section 4.2.1 as an evaluation benchmark, not part of an SFT training set. This inconsistency makes it hard to interpret the scarcity claim; the authors should clarify what Objects365-YorN is, why it has such a high ratio, and whether VSR's inclusion in this table is a labeling error.","section":"Section 3.1.2, Table 2"}],"minor_comments":[{"comment":"The phrase '49% performance gain' is ambiguous; Table 3 shows a 48.8-point absolute increase on What's Up A for the 2B model (44.6 to 93.4), not a 49% relative gain. Please specify whether the reported number is in percentage points or relative percent.","section":"Abstract"},{"comment":"The text says the average accuracy of the 2B and 7B models 'increase by around 9% and 3%.' From Table 3, the 2B average increases by 17.6 points (59.8 to 77.4) and the 7B average by 3.1 points (79.2 to 82.3), so the stated percentages appear inconsistent with the table.","section":"Section 4.5.1"},{"comment":"The CLIP deduplication cutoff of 0.95 and the CLIPScore image-question alignment cutoff of 0.25 are chosen based on manual inspection of only 25 and 100 samples, respectively; a small sensitivity analysis would help establish that the results are not sensitive to these thresholds.","section":"Section 3.3"},{"comment":"The phrase 'the show image' should be 'the shown image'.","section":"Section 3.2.2"},{"comment":"The sentence 'we employ full-string matching on the questions. and CLIP semantic similarity' contains an errant period after 'questions.'","section":"Section 3.3 item 1"},{"comment":"The dataset name 'Pixmo-Cap' is spelled inconsistently; elsewhere it is 'PixMo-Cap'.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know: SpaRE is a useful dataset and pipeline, but the headline '49% gain' on What's Up A is not trustworthy as reported. Training sources include Localized Narratives, which uses COCO images; VSR and What's Up evaluate on COCO images. The paper reports no overlap analysis or image-level deduplication. That is a load-bearing flaw for the central claim.\n\nWhat is actually new: the paper quantifies the long tail of spatial relations in existing VQA datasets (top 17% of relations cover about 90% of examples), and builds a 455k-sample, 3.4M QA-pair dataset from hyper-detailed captions using an LLM, with a thoughtful filtering pipeline. The human evaluation on 400 samples shows ~4% error, reasonable for synthetic data. The fine-tuned models also gain on 3DSRBench and RealWorldQA, which are less likely to overlap with training images, so some genuine transfer is happening.\n\nSoft spots, in order: 1) The overlap issue is serious. Because the QA pairs are synthesized from captions that explicitly name spatial relations for a given image, any COCO image in training could directly give the model the answer for the same image in evaluation. The reported jump of 44.6 to 93.4 on What's Up A (that's 48.8 percentage points, not a 49% relative improvement) could be substantially inflated. 2) The data and code are not yet released; for a data paper, that weakens reproducibility. 3) The '49%' phrasing should be corrected to percentage points in a revision. 4) Minor: the filtering thresholds (CLIP 0.95, CLIPScore 0.25) are manually chosen; that is acceptable but worth noting.\n\nThe citation pattern is on target; they engage with the spatial reasoning literature. The math is straightforward. The pipeline is sound and the data scarcity analysis is a useful contribution.\n\nMy take: this deserves serious peer review, not a desk reject. The flaw is fixable: run exact-match or perceptual-hash deduplication against VSR, What's Up, and 3DSRBench, re-report on disjoint splits, and release the data/code. If the gains shrink, the contribution is still a decent dataset, but not a '49%' story. If the gains hold, it is a solid result. Either way, a good referee will want the overlap analysis before accepting.","headline":"A genuinely useful synthetic spatial-QA dataset, but the headline What's Up gains are suspect until image overlap with training data is ruled out.","tokens_in":19233,"tokens_out":3412,"would_cite":false,"duration_ms":30091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic question-answer pairs distilled from hyper-detailed image captions improve spatial reasoning in vision-language models by up to 49 percentage points on the What's Up benchmark, without hurting general task performance.","keywords":["spatial reasoning","vision-language models","synthetic data","visual question answering","long-tail relations","image captions","fine-tuning","spatial benchmarks"],"falsifier":"Take the test images from VSR and What's Up and split them into those with exact or near-duplicate images in Localized Narratives, DOCCI, or PixMo-Cap and those with no overlap; if SpaRE's advantage over the base model shrinks sharply or disappears on the non-overlapping subset, the claim that the model learned general spatial reasoning is falsified. A second check is to build a held-out spatial benchmark with unseen objects and layouts and measure whether the gain transfers.","tokens_in":18235,"feed_emoji":"🧭","tokens_out":10983,"duration_ms":98650,"temperature":0.7,"pith_summary":"Vision-language models can name objects but often cannot say which object is left, right, above, or facing another. The paper's diagnosis is that their training data is spatially impoverished: in the popular instruction-tuning set analyzed, a handful of relations like left, right, and on dominate while most relations form a tiny long tail. The remedy is a synthetic dataset that turns hyper-detailed human-written image descriptions into 3.4 million spatial question-answer pairs using an instruction-tuned language model, then fine-tunes 2B and 7B VLMs on those pairs. The resulting SpaRE models raise spatial benchmark accuracy by up to 49 percentage points on What's Up A, with several-point gains on VSR, 3DSRBench, and RealWorldQA and no meaningful loss on MMMU, MMBench, or TextVQA. If those gains are genuine rather than memorization of overlapping benchmark images, this is a cheap and scalable recipe for teaching space to VLMs.","feed_headline":"Captions turned into questions lift AI spatial skill 49%","feed_subtitle":"Synthetic QA pairs from image captions improve spatial benchmarks without hurting general tasks.","key_machinery":"The load-bearing mechanism is the caption-to-QA distillation pipeline. Hyper-detailed image captions, typically 100–200 words of human-written description, are first filtered for explicit spatial content, then fed to an instruction-tuned LLM that must return a JSON list of questions and answers restricted to spatial relations such as positions, orientations, distances, layers, boundaries, shadows, overlap, and scale. A cascade of automated checks then removes duplicate questions, references to the description, answers not grounded in the caption, QA pairs whose image-question semantic alignment score is low, and pairs that are not spatial; the surviving 455k samples with 3.4M QA pairs become the training set for SpaRE. The pipeline matters because it converts a relatively small amount of human spatial knowledge, already present in captions, into a large and diverse training signal without rendering synthetic scenes or collecting new annotations.","core_discovery":"The paper's central claim is that the spatial reasoning deficit of VLMs is largely a data-distribution problem, not an architectural one, and that it can be fixed by distilling spatial questions from captions that already describe real images in detail. It first quantifies the scarcity: in the supervised fine-tuning set of a leading open VLM family, the top 17% of spatial-relation types account for over 90% of relation occurrences, leaving rare relations such as facing, opposite, and between almost unseen. It then builds a synthetic dataset of 455k samples and 3.4M QA pairs by prompting an LLM to extract position, orientation, distance, foreground/background, overlap, and scale questions from captions in Localized Narratives, DOCCI, and PixMo-Cap, followed by automated filtering for duplicates, ungrounded answers, non-spatial questions, and image-question mismatch. Fine-tuning Qwen2-VL-2B and Qwen2-VL-7B on this data yields SpaRE models whose average spatial accuracy rises by roughly 9 and 3 points respectively; on What's Up A the 2B model moves from 44.6 to 93.4, and general benchmarks stay stable. The authors interpret this as evidence that synthetic caption-derived QA data narrows the gap between human and VLM spatial reasoning.","pith_inferences":["Beyond the paper: because the training captions come from sources that include COCO, Flickr30k, ADE20K, and Open Images, and two of the evaluation benchmarks use COCO-derived images, part of the headline gain could be image-level memorization; a held-out-image split would separate genuine spatial generalization from familiarity.","Beyond the paper: the pipeline only generates positive assertions about observed relations; adding negative QA pairs that ask whether a wrong relation holds would likely sharpen the contrastive signal and could be tested directly.","Beyond the paper: the same caption-to-QA template could generate other long-tail reasoning skills, such as temporal order or part-whole relations, simply by changing the prompt taxonomy, since the bottleneck identified is data distribution rather than architecture.","Beyond the paper: a controlled test could scramble the spatial phrases in the captions before QA generation; if the gains vanish, the active ingredient is the linguistic description rather than the visual content of the images."],"forward_implications":["Adding more examples of rare spatial relations is likely more valuable than adding more examples of common ones, since the paper shows the distribution is a long tail.","Real-world spatial tasks can be improved without architectural changes: the 2B SpaRE model gains 4.9 points on RealWorldQA.","Spatial fine-tuning can be layered onto existing general-purpose VLMs without sacrificing MMMU, MMBench, or TextVQA performance, making it a safe add-on.","The same caption-to-QA recipe can absorb any future hyper-detailed caption dataset, so the approach scales with data rather than with annotation effort.","The technique does not solve frame-of-reference ambiguity: 3DSRBench gains are smaller, and the paper's own error analysis shows egocentric perspectives remain hard."],"supporting_citations":[{"why":"Supplies Localized Narratives, one of the three hyper-detailed caption sources used to generate QA pairs.","marker":"Pont-Tuset et al., 2020"},{"why":"Supplies DOCCI captions, a key source of fine-grained spatial descriptions.","marker":"Onoe et al., 2024"},{"why":"Supplies PixMo-Cap dense captions, which contribute the largest share of generated QA pairs.","marker":"Deitke et al., 2024"},{"why":"Provides the instruction-tuned LLM that extracts spatial QA pairs from captions.","marker":"Yang et al., 2024"},{"why":"Defines VSR and its relation taxonomy, used both for evaluation and for the long-tail analysis.","marker":"Liu et al., 2023a"},{"why":"Defines the What's Up benchmark where the largest reported gains occur.","marker":"Kamath et al., 2023"},{"why":"Provides the base 2B/7B VLMs that are fine-tuned into SpaRE.","marker":"Wang et al., 2024"},{"why":"Defines the 3DSRBench benchmark that shows smaller gains and remaining 3D limitations.","marker":"Ma et al., 2024"}],"fun_headline_variants":["Synthetic spatial QAs from captions sharpen VLM spatial skills","Turning captions into spatial quizzes boosts VLM benchmarks up to 49%","Rare spatial relations taught via synthetic question-answer data","Mining captions for spatial QA pairs boosts VLM reasoning up to 49%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation benchmarks do not overlap with the training images, so the reported gains reflect learned spatial reasoning instead of the model recognizing pictures it has already seen; the paper does not report an overlap analysis or deduplication between its training sources, which include COCO, Flickr30k, ADE20K, and Open Images, and benchmarks such as VSR and What's Up that use COCO-derived images.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic spatial QAs from captions sharpen VLM spatial skills","Turning captions into spatial quizzes boosts VLM benchmarks up to 49%","Rare spatial relations taught via synthetic question-answer data","Mining captions for spatial QA pairs boosts VLM reasoning up to 49%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4091,"prompt_tokens":1008,"completion_tokens":3083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":3004}},"tokens_in":624,"tokens_out":3083,"duration_ms":19263,"temperature":1.0,"reasoning_tokens":3004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:24:06.161070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the test images from VSR and What's Up and split them into those with exact or near-duplicate images in Localized Narratives, DOCCI, or PixMo-Cap and those with no overlap; if SpaRE's advantage over the base model shrinks sharply or disappears on the non-overlapping subset, the claim that the model learned general spatial reasoning is falsified. A second check is to build a held-out spatial benchmark with unseen objects and layouts and measure whether the gain transfers.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DOCCI captions, a key source of fine-grained spatial descriptions."}],"review_version":1}