{"id":"9fc2be4a-2fde-43f8-825b-47437a61abaa","arxiv_id":"2607.22864","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hierarchical diagnostic benchmark for counting stacked 3D objects shows that MLLMs often reach correct totals without executing the human-like prerequisite sub-tasks, and that training on those sub-tasks markedly improves accuracy.","lead":"This paper introduces Spatial-IQ, a benchmark that breaks the task of counting stacked 3D objects into nine smaller perceptual and spatial-reasoning steps, and uses those steps to diagnose and then train multimodal AI models. Results show that even top AI models can get the final count right without doing the intermediate steps humans use, and that training on those steps substantially improves counting accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Independent sub-task queries cannot support the 'shortcut' inference: failing an isolated sub-task prompt does not show the target computation bypassed that competence.","rationale":"The reader's weakest assumption was that the nine sub-tasks form a causal dependency graph and that failing them implies shortcut behavior. I agree that the causal status of the hierarchy is the load-bearing issue, and I agree that the paper's own data (models reaching correct totals while failing sub-tasks) undermine a strict prerequisite reading. My stress-test sharpens this into a measurement-level gap: because the sub-tasks are queried independently, the observed dissociation cannot distinguish 'the model bypassed the competence' from 'the competence exists but is not elicited by the isolated sub-task prompt.' This is more concrete than the general worry about whether the hierarchy is normative, and it points to a specific control experiment that would settle it. The dataset construction, statistical analysis, and training results are otherwise strong; the human baseline, difficulty controls, and the SFT-CoT/DAPO-tight gains are real evidence. The 32B timeout and the low absolute accuracies noted by the reader are secondary and addressable. Since the reader already recommended CONDITIONAL, my concern does not change the verdict; it adds a specific reason for conditioning the shortcut claim and for requesting the chained-elicitation control before the diagnostic interpretation is presented as established.","tokens_in":43566,"tokens_out":5474,"duration_ms":58337,"concrete_test":"Run a chained-elicitation control on the same 3,000 evaluation scenes for the text free-response models (or at least Qwen and Gemini): present the target Object Counting prompt immediately after asking the model to answer the five hierarchy sub-task questions (Top Layer, Direct Support, Support Column, Visible Object Count, Hidden Object Count) in the stated order, with the final total requested after those answers. Compare Object Counting accuracy in this chained condition against the independent-query condition reported in Table 6. If chained accuracy rises substantially (e.g., by more than the model's 95% confidence interval), the 'succeeds on target while failing sub-tasks' result is partly an artifact of unelicited competence, and the shortcut interpretation must be weakened. If accuracy is unchanged or lower, the bypass inference is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central diagnostic claim is that top models \"often succeed at the target task without succeeding on the lower-level sub-tasks intended to support it, often revealing shortcut behavior.\" The measurement backing this claim is the conditional analysis in Section 5, e.g., Object Counting accuracy conditioned on Visible Object Count and Hidden Object Count correctness (Figure 7, Table 1). But Section 3 states that all sub-tasks and target tasks are queried independently of one another, \"so each task evaluates a competence in isolation rather than chained on prior sub-task answers.\" That design choice creates a logical gap: a model can solve the total-count task through an internal enumeration, 3D reconstruction, or perceptual individuation route that does not match the wording, format, and concept definitions of the isolated \"Visible Object Count\" or \"Hidden Object Count\" prompts. Failure on the independently posed sub-task question therefore reflects failure to answer that specific probe, not demonstrated absence of the underlying competence during the target computation. The label \"shortcut behavior\" is an interpretation, not a measured fact. The human baseline in Figure 7 shows that humans exhibit the hierarchy under the same independent-query protocol, but that establishes a human norm, not a causal necessity for models. The training results in Section 6 provide partial support for the hierarchy as a training signal, yet they compare SFT-CoT (ground-truth sub-task supervision plus a final answer) with SFT-plain (final answer only); this contrast cannot isolate whether the specific hierarchical order and content of the chain is responsible for the gains, as opposed to any additional structured intermediate supervision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Spatial-IQ introduces a hierarchical diagnostic benchmark for spatial intelligence in multimodal LLMs. The target task is object counting in procedurally generated stacks of 3D objects, decomposed into nine sub-tasks (object categorization, cluster/column/layer counts, visible count, top-layer/direct-support/support-column, hidden count) organized into pre-specified dependency relations, plus a mental-rotation probe. Roughly 80,000 scenes are generated in NVIDIA Isaac Sim with analytic per-task ground truth, evaluated in text, multiple-choice, and image-editing formats, and anchored by a human baseline. The paper reports that frontier models reach only 2–18% on the counting target while showing variable sub-task performance, that conditional analyses reveal target-correct responses occurring without correct visible/hidden counts, and that causal claims about 'shortcut behavior' are inferred from these dissociations. A training study on Qwen2.5-VL shows that chain-of-thought supervision over the hierarchy plus RLVR improves target accuracy and hierarchy consistency much more than plain SFT on the final integer.","tokens_in":43809,"tokens_out":7062,"duration_ms":66599,"significance":"If the claims hold, the paper would contribute a useful diagnostic instrument: a large, procedurally generated dataset with voxel-level ground truth, three response formats, a human baseline, and pre-specified hierarchy relations whose statistical treatment (McNemar tests, FDR correction, within-scene lifts) is careful and reproducible. The direct strengths are the analytic ground truths, the difficulty-control validation, the format-stable Internal Referential Chain, and the training result showing that decomposition-aware supervision outperforms final-answer-only SFT. However, the central headline—that models 'often succeed at the target task without succeeding on the lower-level sub-tasks'—is overstated relative to the raw accuracies, and the inference from independent sub-task probes to 'shortcut behavior' is an interpretive leap rather than a measured fact. The benchmark remains valuable as a diagnostic tool for dissociations under isolated probes, with the shortcut/causal language needing substantial revision or additional chained-query evidence.","major_comments":[{"comment":"The headline claim that top-performing models \"often succeed at the target task without succeeding on the lower-level sub-tasks\" is not supported as stated. The best model reaches 17.7% on Object Counting (Table 1, Fig. 4), so target-task success is not frequent in absolute terms. The data do support a conditional claim: among the small set of target-correct responses, a substantial fraction occur with incorrect Visible or Hidden Object Count responses (e.g., for Gemini, about 56% of target-correct trials come from cells where Visible Object Count is wrong). As written, the absolute-frequency reading overstates model competence and should be reworded to \"when models succeed at the target, they often fail the isolated sub-task probes\" or to a conditional formulation.","section":"Abstract; §5, Fig. 7, Table 1"},{"comment":"Because all sub-tasks and target tasks are queried independently of one another, the conditional analyses show only that a model that fails an isolated sub-task query can still answer the target query correctly. They do not demonstrate that the target computation bypassed the sub-task competence; a model could use an internal enumeration, 3D reconstruction, or perceptual-individuation route that is not captured by the wording or format of the isolated sub-task prompt. The label \"shortcut behavior\" in the abstract and Section 7 is therefore an interpretation rather than a measured fact. I recommend either adding a chained-query condition (feeding model- or ground-truth-generated sub-task answers into the target prompt) or replacing \"shortcut behavior\" with \"dissociation under independent probes\" throughout the paper.","section":"§3, §5, Fig. 7"},{"comment":"The hierarchy is introduced as a \"causal dependency graph\" in which sub-tasks are prerequisites for the target task. This is a normative commitment drawn from developmental psychology, not an empirical fact about either humans or machines. The paper's own data—models passing the target while failing prerequisites—are equally compatible with the hierarchy being an arbitrary decomposition for these models. The authors should either test the causal reading directly (for example by manipulating sub-task competence through training or by fitting a hierarchical diagnostic model) or explicitly restrict their conclusions to the pre-specified ordering, presenting it as a hypothesis rather than a demonstrated causal chain.","section":"§3, Fig. 3, Appendix C"},{"comment":"The scene-validity convention removes any hidden object that does not support a visible object, so Hidden Object Count and the hidden branch of Object Counting concern only support-required occlusion. The abstract's general claim about counting \"including those occluded from view\" is broader than what the benchmark actually measures. This is a defensible design choice for unambiguous ground truth, but it should be stated prominently in the main text and abstract so readers do not generalize the results to arbitrary occlusion or to hidden objects that are not physically required for support.","section":"Appendix A, Definition 6; Appendix D"},{"comment":"The training experiment demonstrates that CoT supervision over the benchmark's own ground-truth labels improves performance on that benchmark, but it is a single-family (Qwen2.5-VL), single-modality (text) demonstration. Because SFT-CoT and DAPO-tight are trained and evaluated on the same task distribution, the large gains on hierarchy columns may partly reflect learning the exact sub-task answer format rather than a genuinely general spatial competence. The out-of-distribution object-category result is encouraging, but it does not establish that the decomposition transfers to other spatial task families. I would ask the authors to label this as a demonstration for the Spatial-IQ task family rather than a general training principle, and to report the missing 4% of the 32B DAPO-tight evaluation more explicitly.","section":"§6, Appendix H.3"}],"minor_comments":[{"comment":"The figure labels contain the artifact \"VLA/uni00AD0\" in the model ordering; this should be corrected to \"VLA-0\".","section":"Figures 2, 4"},{"comment":"Appendix B does not report the number of annotators or the number of human responses per task, and it is unclear whether the human row in Table 6 uses the same n=3000 scenes or a smaller human-only sample. Please state the human sample size explicitly and use it in the Wilson confidence intervals.","section":"Appendix B, Table 6"},{"comment":"The difficulty-calibration procedure was tuned on pilot frontier models; this selection information is important for interpreting the benchmark and should be summarized in the main text near the dataset description rather than only in an appendix.","section":"Appendix D"},{"comment":"The explanation of Qwen's negative chance-adjusted Hidden Object Count artifact is asserted but not accompanied by the supporting data; please include the quantitative evidence or remove the claim.","section":"Appendix F.2"},{"comment":"The text refers to \"accompanying configuration files\" for hyperparameters, but these are not included in the manuscript; please provide them in the supplement so the training conditions are fully reproducible.","section":"Appendix H.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical benchmark paper with careful statistics and a useful dataset, but the headline is over-sold relative to the evidence. The core issue is the gap between independent-probe dissociations and the paper's \"shortcut behavior\" and causal-hierarchy language. I would not reject on novelty or ethics grounds; the work is within scope and the training result is interesting. The revision should either add a chained-query experiment that can support the shortcut claim or systematically soften the causal and shortcut language to match the measurement design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about Spatial-IQ. First, the benchmark construction is genuinely solid: roughly 80,000 procedurally generated 3D scenes with voxel-derived ground truth, three response formats, a human baseline, and coherent difficulty controls. The nine sub-tasks under object counting are the first decomposition of a single target into dependency-linked sub-tasks with per-task ground truth, and the human results show the expected hierarchy under the same protocol. Second, the training result is real and practically significant: CoT supervision over the hierarchy plus RLVR lifts Qwen2.5-VL-32B from 2.9% to 62.6% on the counting task, and SFT-plain collapses on every hierarchy column while reaching similar final accuracy. That dissociation is the strongest evidence in the paper that the hierarchy carries useful training signal.\n\nThe soft spots are in the interpretation, not the construction. All sub-tasks and target questions are queried independently, so failing an isolated \"how many visible objects?\" prompt does not show that the target computation bypassed that competence. A model could answer the total through a different formulation of the same underlying ability. The \"shortcut behavior\" label is an interpretation, not a measured fact. The stress-test note lands: the causal reading is undercut by the paper's own data, where models reach correct totals while failing the supposed prerequisites. Absolute accuracies are low (best model 17.7%), so \"often succeed at the target task\" is misleading. The causal dependency graph is imported from developmental psychology; the paper hedges with \"hypothesized,\" but the abstract and discussion lean on it more heavily. The SFT-CoT versus SFT-plain comparison also cannot isolate whether the specific order and content of the chain matters versus any structured intermediate supervision.\n\nTo the authors' credit, they acknowledge key limitations: cross-modality non-equivalence, single-family training, and the 32B evaluation timeout at 96% completion. Statistics are proper throughout (McNemar, FDR, Wilson intervals). This is a useful resource for anyone working on MLLM evaluation or spatial reasoning, and I would cite it for the benchmark and the training result even without buying the full shortcut interpretation.\n\nThe paper deserves a serious referee. I would recommend conditional acceptance with a request to reframe the central claim as \"models do not reproduce the human dependency structure under independent probes,\" and to either add a chained evaluation where sub-task outputs feed the target or to drop the causal language. As written, the measurement contribution is strong; the diagnostic conclusion needs rewording.","headline":"Well-built spatial diagnostic whose headline 'shortcut' finding outruns the evidence from its own independent-query design.","tokens_in":44411,"tokens_out":4059,"would_cite":true,"duration_ms":38164,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counting boxes without the parts: a spatial hierarchy exposes shortcut behavior in vision models","keywords":["spatial intelligence","hierarchical benchmark","object counting","multimodal large language models","chain-of-thought","reinforcement learning with verifiable rewards","3D structure","occlusion reasoning"],"falsifier":"Train a model on total counts only, then compare its Visible × Hidden conditional matrix to a decomposition-trained model's matrix: if the endpoint-only model matches the decomposition-trained model's row and column pattern (highest in both-correct, lowest in both-wrong), the matrix signature would not diagnose shortcut behavior. Conversely, if scenes with zero hidden objects are solved by a model while matched hidden-heavy scenes fail, the hidden-inference step is load-bearing.","tokens_in":43352,"feed_emoji":"🧊","tokens_out":3280,"duration_ms":26449,"temperature":0.7,"pith_summary":"This paper argues that current multimodal large language models do not solve spatial tasks like counting boxes in a stack the way humans do: they reach the right total without going through the lower-level perceptual and cognitive steps that produce the total. The authors introduce Spatial-IQ, a diagnostic framework that decomposes 3D object counting into nine sub-tasks arranged in a hierarchy, and show that every frontier model fails many of those sub-tasks even when its final count is correct. They then show that training a model on the hierarchy itself, by asking it to output the sub-task answers in order before the total, plus reinforcement learning on verifiable rewards, sharply improves both intermediate and final accuracy. If right, this means a spatial benchmark must report more than a final accuracy score, and that decomposing a skill into its human-developmental prerequisites is itself a useful training signal.","feed_headline":"Counting boxes without the parts","feed_subtitle":"A 9-step spatial hierarchy exposes shortcut behavior in vision models and trains it away.","key_machinery":"The central machine is a pre-specified causal hierarchy over one target task: counting objects in a gravity-supported 3D stack. The hierarchy has nine sub-tasks (object categorization, cluster count, column count, layer count, visible object count, top layer, direct support, support column, hidden object count) joined by four testable relations: the Visible Count Hierarchy, the Hidden Count Hierarchy, the Internal Referential Chain (top layer to direct support to support column), and the Summation Mechanism (visible count + hidden count = total). The data are ~80,000 procedurally generated scenes in Isaac Sim, with per-task ground truth derived from voxel control; hidden objects are defined by a depth-buffer test that keeps an occluded object only if it supports a visible object. The paper evaluates models in three output formats and a human baseline, and the training phase uses the hierarchy as a chain-of-thought template plus RLVR (DAPO) with a gated integer-distance reward.","core_discovery":"The Spatial-IQ framework demonstrates that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure.","pith_inferences":["The observable signature of human-like spatial reasoning in this framework is the Summation Mechanism's matrix (both visible and hidden correct well above both wrong); a model that reproduces that matrix without training on the hierarchy would still satisfy the diagnostic even if it never explicitly sums its two sub-answers.","The results generalize to other spatially decomposable tasks (paper folding, water-level prediction, perspective taking) by instrumenting the same pattern: identify prerequisites, encode them as a chain of tasks, and reward correct intermediate steps, an extension the paper itself points to.","The hidden-object support rule encodes a modeling commitment: hidden objects are defined by a support-based occlusion convention rather than any occluded volume, so hidden count is a logical inference about gravity support rather than a purely perceptual occlusion fact, and is the most likely place a competing benchmark would depart.","A direct next test is whether the SFT-CoT/DAPO gains persist under viewpoint, layout, and texture perturbations beyond the categories already tested, since training used one object category and evaluations on others were held out."],"forward_implications":["Benchmarks should report sub-task or component accuracy, not just final accuracy: two models with identical total counts can differ by 30+ points on the intended mechanism.","Chain-of-thought supervision over a spatial sub-task hierarchy plus RLVR moves a Qwen2.5-VL-32B from 2.9% to 62.6% Object Counting accuracy, closing much of the gap to the 82.1% human baseline on this benchmark.","A model trained on the final answer alone reaches the same total count as a decomposition-trained model at 7B scale, but collapses to zero on every hierarchy relation, so endpoint-only training teaches a shortcut that does not preserve the intended competence.","The trained model generalizes to unseen object categories with only a small drop, indicating the hierarchy encodes a general spatial competence rather than a memorized cube-specific answer.","The full hierarchical decomposition is specific to Object Counting; Mental Rotation behaves as an independent spatial probe that does not share the same sub-task chain."],"supporting_citations":[{"why":"Supplies the block-counting subtest of the Kaufman Assessment Battery for Children, the standardized human task that motivates the target Object Counting task.","marker":"[17]"},{"why":"Provides the Piaget-Inhelder developmental-stage framework that organizes the nine sub-tasks into a principled ordering.","marker":"[27]"},{"why":"SpaCE-10, the compositional spatial benchmark this work contrasts with and extends by testing prerequisite atomic capabilities.","marker":"[11]"},{"why":"Supplies the DAPO RLVR recipe used in the two-phase training regime.","marker":"[46]"},{"why":"Provides the chain-of-thought step-verification approach that the hierarchical CoT supervision builds on.","marker":"[22]"},{"why":"Replicator toolkit used for ground-truth image renderings and the depth-buffer occlusion test that defines hidden objects.","marker":"[25]"},{"why":"Tatsuoka's rule-space method, the attribute-hierarchy convention in cognitive diagnosis that motivates pre-specifying prerequisite structures.","marker":"[37]"},{"why":"Hierarchical diagnostic classification models, the statistical family used to test whether the pre-specified attribute hierarchy is empirically preserved.","marker":"[39]"}],"fun_headline_variants":["Vision models pass counting by skipping spatial sub-steps","Spatial-IQ: models hit target but miss the building blocks","9-step spatial test reveals shortcut reasoning in MLLMs","Hierarchical test: models count boxes without true spatial logic","Shortcut alert: models pass counting, fail sub-tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nine sub-tasks are not just useful labels but a causal dependency chain for object counting, so that a model which reaches the correct total without completing that chain is taking a shortcut rather than using another legitimate strategy.","fun_headline_variants_meta":{"raw":{"variants":["Vision models pass counting by skipping spatial sub-steps","Spatial-IQ: models hit target but miss the building blocks","9-step spatial test reveals shortcut reasoning in MLLMs","Hierarchical test: models count boxes without true spatial logic","Shortcut alert: models pass counting, fail sub-tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1412,"prompt_tokens":878,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":450}},"tokens_in":494,"tokens_out":534,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:27:46.867338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on total counts only, then compare its Visible × Hidden conditional matrix to a decomposition-trained model's matrix: if the endpoint-only model matches the decomposition-trained model's row and column pattern (highest in both-correct, lowest in both-wrong), the matrix signature would not diagnose shortcut behavior. Conversely, if scenes with zero hidden objects are solved by a model while matched hidden-heavy scenes fail, the hidden-inference step is load-bearing.","supporting_citations":[{"cited_title":"Kaufman assessment battery for children.Psychological Assessment, 1983","cited_arxiv_id":null,"evidence_quote":"Supplies the block-counting subtest of the Kaufman Assessment Battery for Children, the standardized human task that motivates the target Object Counting task."},{"cited_title":"Basic Books, New York, 1969","cited_arxiv_id":null,"evidence_quote":"Provides the Piaget-Inhelder developmental-stage framework that organizes the nine sub-tasks into a principled ordering."},{"cited_title":"Replicator tutorials — omniverse isaac sim 4.2.0 documentation","cited_arxiv_id":null,"evidence_quote":"Replicator toolkit used for ground-truth image renderings and the depth-buffer occlusion test that defines hidden objects."},{"cited_title":"Rule space: An approach for dealing with misconceptions based on item response theory.Journal of educational measurement, pages 345–354, 1983","cited_arxiv_id":null,"evidence_quote":"Tatsuoka's rule-space method, the attribute-hierarchy convention in cognitive diagnosis that motivates pre-specifying prerequisite structures."},{"cited_title":"Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies.Psychometrika, 79(2):317–339, 2014","cited_arxiv_id":null,"evidence_quote":"Hierarchical diagnostic classification models, the statistical family used to test whether the pre-specified attribute hierarchy is empirically preserved."}],"review_version":2}