{"id":"87335d89-e1f8-4056-9434-d84b24dffaba","arxiv_id":"2505.20728","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current vision-language models fall far short of humans on spatial reasoning, especially when they must generate answers directly instead of choosing from options.","lead":"The paper introduces Jigsaw-Puzzles, a benchmark of 1,100 real-world images with five spatial reasoning tasks. It evaluates 24 vision-language models and finds that even the best model scores 77% overall and only 30% on an open-ended order generation task, while humans score above 90%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline Order Generation figure (30.00%) rests on an undocumented exact-match parser for free-form outputs; without normalization rules, equivalent correct sequences may be scored as wrong, inflating the claimed human-VLM gap.","rationale":"The reader's weakest assumption (small/non-naive human sample) is legitimate but not the most load-bearing: the model-human gap is so large (64 points on Order Generation; 20+ overall) that even a 10-15 point correction to the human baseline would not overturn the qualitative conclusion. By contrast, the 30.00% Order Generation number is the paper's sharpest and most repeated evidence, and it is a model-side measurement whose validity depends on an undocumented parser. In open-ended sequence tasks, exact-match scoring without normalization is a known source of large underestimation. The appendix does not include the parsing code; the only stated metric is exact match. A single re-scoring experiment can settle whether the headline number is real. The verdict stays CONDITIONAL because the benchmark and the core finding are plausible, but the missing parser documentation and the human-baseline details both need to be addressed before the quantitative claims are relied upon.","tokens_in":11943,"tokens_out":6118,"duration_ms":70104,"concrete_test":"Obtain the raw Order Generation outputs for Gemini-2.5-Pro (and at least the next two best models) on Jigsaw-Puzzles-Lite, and re-score them with a lenient normalizer that strips whitespace/punctuation, canonicalizes patch identifiers (e.g., maps 'A/B/C/D' to 1/2/3/4 and spatial descriptions to positions), and accepts any unique permutation matching the ground-truth order. Also manually label 20 raw outputs to confirm equivalence classes. If the re-scored accuracy stays near 30.00%, the parsing concern is not the driver; if it rises materially (e.g., above 50%), the paper's central gap is overstated and must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'even the strongest model ... Order Generation ... only 30.00%, far below human 94.09%' (Abstract; Fig. 8, §4.2) depends entirely on how open-ended model outputs are scored. §4.1 defines the metric only as 'exact match accuracy (%)', and Appendix B supplies the prompt template (Fig. 18) but no answer parser or normalization rule. For a sequence-generation task, a model may state the correct permutation as 'A B C D', 'top-left, top-right, bottom-left, bottom-right', '1 2 3 4', or with punctuation; naive exact-string matching would mark all but one of these equivalent answers wrong. The paper gives no evidence that the released evaluation handles this. Thus 30.00% is a lower bound on demonstrated competence, not a measured ceiling. This is more load-bearing than the human-sample issue: even a substantially lower human baseline (e.g., 80%) would leave a large gap, whereas a weak parser could move the model score by tens of points and change the comparative headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Jigsaw-Puzzles, a new benchmark for evaluating spatial reasoning in vision-language models, built from 1,100 real-world images with five automatically generated tasks (Missing Piece Selection, Piece Localization, Connection Verification, Anomaly Detection, Order Restoration) and an open-ended Order Generation variant on a 220-image subset. The authors evaluate 24 proprietary and open-source VLMs, report human performance from three participants on the subset, and find that the best model (Gemini-2.5-Pro) reaches 77.14% overall but only 30.00% on Order Generation, far below the human performance above 90%. They also analyze task correlations, scaling with model size, and the effect of reasoning-enhanced models.","tokens_in":12128,"tokens_out":3962,"duration_ms":35638,"significance":"The benchmark addresses a real gap: existing spatial reasoning benchmarks are largely synthetic or multiple-choice, and the paper provides a scalable, automated pipeline with visual complexity from real images. The task hierarchy from perception to multi-step reasoning is well motivated, and the evaluation spans a wide range of current VLMs. The Order Generation task is a valuable attempt to move beyond option-constrained evaluation. The main results would be significant if confirmed, because they quantify a persistent VLM-human gap in open-ended spatial reasoning. However, the headline figures depend on evaluation details (answer parsing and human baseline) that are not currently documented.","major_comments":[{"comment":"The Order Generation result of 30.00% (Gemini-2.5-Pro) in Figure 8 is the paper's headline evidence for a VLM-human gap in open-ended spatial reasoning, but the paper specifies only 'exact match accuracy (%)' as the metric and does not describe the answer parser or normalization rules. For a free-form sequence task, correct answers can be expressed as 'A B C D', 'A, B, C, D', '1 2 3 4', or with piece names/descriptions; naive exact-string matching would mark all but one of these equivalent answers wrong. The authors need to provide the parsing and normalization procedure, release the evaluation code, or show example model outputs with their scoring outcomes. As written, 30.00% is a lower bound on demonstrated competence rather than a measured accuracy, and this directly affects the central comparative claim.","section":"§4.1, §4.2, Appendix B (Fig. 18)"},{"comment":"The human baseline is derived from only three participants on a 220-image subset, and the paper does not state whether the participants were naive or involved in dataset curation. If the participants are the authors or otherwise familiar with the design, the 96.36% overall and 97.73% Order Restoration accuracies could overstate the human-VLM gap. In addition, no inter-participant variance or confidence intervals are reported, so the precision of the human upper bound is unknown. The authors should describe participant recruitment and naivety status and report per-participant scores.","section":"§4.1 (Human Performance), Table 3"},{"comment":"All VLM scores are reported as point estimates from a single run without error bars or confidence intervals, and the correlation and scaling analyses in Figures 5 and 6 use only 24 model-level points without significance tests or confidence bounds. Because several model differences in Tables 2 and 3 are only a few percentage points on samples of 1,100 or 220 items, some of the qualitative conclusions (e.g., the ordering of top-performing models, or the strength of task correlations) may not be robust. The paper should provide at least bootstrap confidence intervals for the main accuracies and p-values for the correlations.","section":"§4.2 (Main Results), Tables 2-3, Figures 5-6"}],"minor_comments":[{"comment":"The sentence 'As shown in Tables 2,' should be 'As shown in Table 2,' and 'Table 2, 3 report' should be 'Tables 2 and 3 report' for grammatical correctness.","section":"§4.2"},{"comment":"The caption says 'Evaluation of Order Restoration and Order Generation tasks on Jigsaw-Puzzles-Lite'; since Order Generation is introduced only on the Lite subset, clarify that Order Restoration results are also on Lite in this figure, whereas Table 2 reports the full set.","section":"Figure 8 caption"},{"comment":"The 'p-value-based critical value' is defined as the minimum accuracy to outperform random guessing at p=0.05, but the formula or exact computation is not shown; include the calculation so readers can reproduce the 32.56% overall threshold.","section":"§4.1 (Baselines)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim hinges on the 30.00% Order Generation figure, which is not interpretable without the answer parser. I strongly recommend requiring the authors to release the evaluation code and parse rule as a condition of acceptance. If the parser turns out to be overly strict, the headline result could shift by tens of points, changing the paper's main message. The small human sample is also a concern, but it is secondary to the parser issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark itself is genuinely useful: 1,100 real-world images, five puzzle tasks spanning perception to multi-step reasoning, and an open-ended Order Generation setting that removes answer options. That last piece is the real contribution, and the finding that even Gemini-2.5-Pro drops from 77% overall to 30% on generation is striking. The paper also evaluates 24 models and releases the dataset and construction scripts, which is the right way to do benchmark work.\n\nThe main soft spot is the evaluation of Order Generation. The metric is defined only as exact match accuracy, and there is no description of how free-form model outputs are parsed or normalized. A model might output the correct sequence as 'A B C D', '1 2 3 4', 'top-left, top-right, bottom-left, bottom-right', or with punctuation. Naive string matching would mark all but one of those wrong. Without a documented normalization rule, the 30.00% figure is a lower bound on demonstrated competence, not a measured ceiling. That is load-bearing because the headline human-VLM gap on generation depends on it. The fix is straightforward: ship the parser, describe the normalization steps, and ideally report a human-annotation-based re-score on a sample.\n\nThe human baseline issue is real but secondary. Three participants on 220 images is small, and the paper does not say whether they were naive or involved in curation. Even if they were the authors, the gap would remain large, so this is a disclosure problem rather than a fatal one. Still, report variance and state participant background.\n\nTwo smaller points: the 'fully automated construction' claim is overstated given the expert filtering step, and the correlation/scaling analyses on 24 models are suggestive but should not be over-interpreted. The limitations section is honest about scope.\n\nThis paper deserves a serious referee. The core direction holds up, and the generation task is a legitimate diagnostic advance. I would send it to review with a request for clarification on parsing, human participant details, and confidence intervals. I would likely cite it when discussing spatial reasoning benchmarks, and it is worth bringing to our reading group once it is in camera-ready form.","headline":"Useful spatial-reasoning benchmark with a plausible headline result, but the 30% Order Generation figure needs a documented parser before I'd trust it.","tokens_in":12661,"tokens_out":1896,"would_cite":true,"duration_ms":22883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"New jigsaw benchmark shows best vision-language model trails humans by 19 points.","keywords":["spatial reasoning","vision-language models","benchmark","jigsaw puzzles","multi-step reasoning","order restoration","human performance baseline","real-world image evaluation"],"falsifier":"Recruit at least twenty participants with no prior exposure to Jigsaw-Puzzles and have them complete the full Lite subset under the same no-tools conditions; if their average accuracy lands near the VLM level rather than 96%, the claimed human-VLM gap does not hold.","tokens_in":11745,"feed_emoji":"🧩","tokens_out":5472,"duration_ms":50881,"temperature":0.7,"pith_summary":"The paper introduces Jigsaw-Puzzles, a benchmark of 1,100 real-world images that turns jigsaw-style tasks into five evaluation levels: selecting a missing piece, localizing pieces, verifying adjacency, detecting anomalies, and restoring shuffled order. The authors claim that these tasks trace the human cognitive path from seeing to understanding to reasoning, and that current vision-language models fail at the reasoning end. Across 24 models, the strongest tested model reaches 77.14% overall accuracy while human participants average 96.36%; on the open-ended Order Generation task the same model drops to 30.00%, against 94.09% for humans. The paper argues that existing spatial-reasoning benchmarks rely on synthetic or simple scenes, and that a real-image, automatically constructed benchmark is needed to expose this gap and guide progress.","feed_headline":"Best VLM scores 77% on spatial puzzles; humans score 96%","feed_subtitle":"Even top models collapse when they must rebuild the order of shuffled pieces without answer choices.","key_machinery":"The load-bearing mechanism is the five-task cognitive ladder itself, built from a fully automated template pipeline that produces deterministic question-answer pairs from real photographs. Each task is designed to isolate one stage, from perception (Missing Piece Selection) through single-step structural reasoning (Piece Localization, Connection Verification, Anomaly Detection) to multi-step reconstruction (Order Restoration), so that the accuracy drop between adjacent rungs diagnoses where spatial reasoning breaks. A second mechanism, the Jigsaw-Puzzles-Lite subset with an Order Generation task that removes answer options, turns the benchmark from a perception test into a test of autonomous reasoning.","core_discovery":"On its own terms, the paper's central discovery is that VLM spatial reasoning is not uniformly weak but collapses as the cognitive demand rises: models approach human accuracy on missing-piece selection, show emerging competence on single-step tasks such as localization and connection verification, and fall far behind on multi-step order restoration. The gap is not a perception failure, since the strongest model reaches 97.18% on the hard missing-piece condition, but a failure to compose local spatial cues into a reconstructed whole. The paper also finds that reasoning-enhanced models improve most on multi-step tasks, apparently because answer choices let them re-evaluate and self-correct, and that performance correlates with model scale and with basic spatial-understanding ability. When the multiple-choice scaffold is removed in Order Generation, the best model drops from 73.64% to 30.00%, which the paper reads as evidence that current VLMs cannot autonomously build a coherent spatial reasoning chain.","pith_inferences":["My reading: the benchmark may partly measure a model's ability to exploit low-level cues such as patch borders, seams, and color continuity rather than true spatial structure; a controlled version with seamless cuts would test this.","My reading: the three-participant human baseline makes the headline 19-point gap fragile; a larger naive sample is needed before treating 96% as the upper bound.","My reading: the same template pipeline could be extended to 3D scenes, temporal sequences, or video jigsaws, giving a direct test of whether the reasoning deficit is specific to static 2D layout."],"forward_implications":["If the gap is real, claims of robust VLM spatial intelligence should be treated as unsupported until a model substantially closes the 96% human baseline on this benchmark.","The correlation between Missing Piece Selection and reasoning tasks implies that improving basic spatial understanding may be the most direct route to better multi-step VLM reasoning.","Benchmark builders should prefer real-image, automatically generated tasks over synthetic grids, since the failure pattern is only visible under high visual complexity.","Multiple-choice formats can mask reasoning failures, so open-ended variants such as Order Generation should become standard diagnostic checks.","Model scaling alone is unlikely to close the gap, since even the largest proprietary models remain far behind humans on multi-step tasks."],"supporting_citations":[{"why":"Supplies the CC3M image pool that the benchmark filters into 1,100 real-world photos.","marker":"(Sharma et al., 2018)"},{"why":"CLIP similarity is used to select hard distractors in Missing Piece Selection.","marker":"(Radford et al., 2021)"},{"why":"Existing spatial benchmark used to show that prior evaluations lack real-world complexity.","marker":"(Pothiraj et al., 2025)"},{"why":"Existing spatial-relations benchmark whose synthetic setting motivates Jigsaw-Puzzles.","marker":"(Stogiannidis et al., 2025)"},{"why":"Visual grid-puzzle benchmark compared as evidence of task constraints in prior work.","marker":"(Ren et al., 2025)"},{"why":"LEGO-Puzzles multi-step spatial benchmark compared against the new tasks.","marker":"(Tang et al., 2025)"},{"why":"Cognitive study motivating jigsaw puzzling as a multi-stage testbed.","marker":"(Fissler et al., 2018)"},{"why":"Supplies the Pearson-correlation method used to show that spatial understanding shapes reasoning.","marker":"(Zhang et al., 2025)"}],"fun_headline_variants":["Spatial puzzles: VLMs drop to 30% when choices vanish","Best VLM lags humans by 19 points on spatial reasoning","Order generation stumps top VLM at 30% vs human 96%","Jigsaw-Puzzles benchmark reveals VLM order-reasoning gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human upper bound rests on three participants solving a 220-image subset, and the paper does not say whether they were naive to the benchmark or involved in building it.","fun_headline_variants_meta":{"raw":{"variants":["Spatial puzzles: VLMs drop to 30% when choices vanish","Best VLM lags humans by 19 points on spatial reasoning","Order generation stumps top VLM at 30% vs human 96%","Jigsaw-Puzzles benchmark reveals VLM order-reasoning gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3036,"prompt_tokens":957,"completion_tokens":2079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":573,"tokens_out":2079,"duration_ms":15728,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:47:03.343143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit at least twenty participants with no prior exposure to Jigsaw-Puzzles and have them complete the full Lite subset under the same no-tools conditions; if their average accuracy lands near the VLM level rather than 96%, the claimed human-VLM gap does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cognitive study motivating jigsaw puzzling as a multi-stage testbed."}],"review_version":1}