{"id":"5d812e54-9e5a-40b6-a44c-8e5349a4f162","arxiv_id":"2502.04359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On the CopsRef dataset, referring expression comprehension reveals that vision-language models fail at dynamic spatial relations, multi-relation expressions, and negations, while task-specific models handle geometric relations better.","lead":"This paper tests how well three vision-language models ground spatial language by having them find objects described by complex referring expressions, such as 'the branch to the right of the log that is behind the bear.' It finds the models struggle with dynamic relations, multiple spatial relations, and negations, while performing better on simple geometric terms like left and right.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline negation finding is an artifact of selection bias: §5.5 restricts to expressions where all models already fail, so it cannot establish that models struggle with negated spatial relations or compare their failure rates.","rationale":"The reader's CONDITIONAL verdict is fair. The proposal to use REC rather than captioning/VQA is well-motivated, and Table 5's category-wise accuracies are a useful contribution. What has to be true for the central claim to hold is that each advertised source of difficulty (visual ambiguity, multi-relation composition, negation) is measured without conditioning on the outcome being explained. The negation analysis is the least secure: the test split in §5.5 is literally constructed from expressions where every model fails, so the paper cannot distinguish 'models struggle with negation' from 'models struggle with these hard examples.' The same selection problem does not apply to the category-wise or multi-relation analyses, which use full splits. The visual complexity split is also vulnerable because the labeling detector is one of the evaluated models; this reduces confidence in Table 8 but is not definitional. The Kendall tau z-scores used n=6 rather than n=12 in Eq. 1; recomputing with n=12 makes the GDINO–LLaVA correlation stronger (z≈3.30), so that conclusion is unchanged. Verdict stays CONDITIONAL, with the negation re-analysis as the primary requested condition.","tokens_in":15556,"tokens_out":10103,"duration_ms":97076,"concrete_test":"Recompute §5.5 from the full CopsRef test set, not the all-fail subset: extract every expression containing 'not' (1 and 2 occurrences), compute per-model IoU>0.5 accuracy on that full set, and compare against a control sample of non-negated expressions matched on number of spatial relations, expression length, and object/attribute composition. If full-set negation accuracy is not significantly lower than the control, the 'all models struggle with negated spatial relations' claim fails. Also report the proportion of negated expressions for which all three models fail, to calibrate Table 9.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest of the paper's platform claims—that REC exposes model struggles with negation—rests on a selection-bias artifact. Section 5.5 first restricts to the 109 expressions (73 with two 'not's, 36 with one) for which all three models already produce IoU<0.5, then reports how often each model 'fails to recognize at least 1 negation' within that subset. Because the subset is defined by total failure, the rows of Table 9 cannot estimate the prevalence of negation failure, the relative difficulty of one vs two negations, or which model is best; every claim of the form 'model X has the highest number of failure instances' is conditional on a set chosen for universal failure. The missing denominators are the total number of negated expressions in CopsRef and matched non-negated expressions with comparable length and spatial complexity. Without those, the advertised negation capability of the REC platform and headline finding (5) are unsupported. A parallel circularity exists in the visual complexity split (§4.2), where Grounding DINO labels instance counts and is then scored on that split, but the negation analysis is the more direct threat to a headline conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Referring Expression Comprehension (REC) as a platform for diagnosing spatial reasoning in vision-language models. Using the CopsRef test set, the authors categorize 51 spatial relations into 8 semantic classes, split expressions by number and type of spatial relations (none, one, two-chained, two-and, two-or, three), and evaluate MGA-Net, Grounding DINO, LLaVA, and an OWL-ViT label-only baseline. The headline findings are that spatial relations often improve grounding relative to attribute-only expressions; increasing relation count hurts zero-shot VLMs but not the task-specific MGA-Net; directional relations are hardest; task-specific models and VLMs differ on geometric versus proximity relations; and all models struggle with negated relations. The paper also reports a visual-complexity split and qualitative case studies.","tokens_in":15786,"tokens_out":5394,"duration_ms":50956,"significance":"If the analysis were sound, the paper would make a useful contribution as a fine-grained evaluation of spatial grounding along several dimensions not jointly covered by prior VQA/captioning work. The authors are careful to report multiple runs, disclose training constraints, make their evaluation splits explicit, and include an appendix documenting models that failed to produce usable outputs (InstructBLIP, OpenFlamingo). These are genuine strengths. The per-category accuracy measurements in Tables 4-7 are straightforward and likely reproducible. However, the two novel analysis axes most emphasized in the abstract and findings—negation handling and visual complexity—rest on methodological choices that undermine the corresponding conclusions. The central platform claim for negation (finding 5) is not supported by the present design; the visual-complexity result is partly self-referential for Grounding DINO; and the architecture-level interpretation of MGA-Net's stability is confounded with training on the evaluation set.","major_comments":[{"comment":"The negation analysis is selection-biased. The authors first restrict to expressions with one or two 'not's for which all three models already have IoU < 0.5 (36 and 73 expressions), then report how often each model 'failed to recognize at least 1 negation' within that subset. Because every expression in the subset is already a failure for all models, the rows cannot estimate the prevalence of negation errors, the relative difficulty of one vs. two negations, or which model is better. In particular, the statement that 'Grounding DINO has the highest number of failure instances' and the conclusion that 'LLaVA handles negations better' are artifacts of conditioning on universal failure. To support finding (5), the paper needs denominators over all negated expressions in CopsRef and a matched non-negated control set (e.g., expressions with comparable length, entities, and attributes).","section":"§5.5, Table 9"},{"comment":"The visual-complexity split is constructed by using Grounding DINO to count instances of entities extracted by spaCy, and the same model is then scored on that split. Any systematic over- or under-counting by Grounding DINO, or errors in spaCy entity extraction, directly contaminates the single- vs. multi-instance label and, for the Grounding DINO row, makes the comparison partly self-referential. The authors should validate the instance counts against human annotation or an independent detector, and either report results excluding the model used for splitting or analyze sensitivity to the label set.","section":"§4.2 and Table 8"},{"comment":"The claim that MGA-Net's compositional architecture explains its stable accuracy across zero, one, two, and three spatial relations is confounded: MGA-Net was trained on CopsRef (for ten epochs, after being initialized from a detector), whereas Grounding DINO, LLaVA, and OWL-ViT were evaluated zero-shot. Better performance on the training distribution is an alternative explanation that is not controlled for. At minimum, the paper should temper the architectural conclusion and discuss this confound in Section 5.3.","section":"§5.3, Table 7 (RQ4)"},{"comment":"Finding (1), that spatial relations improve grounding relative to attributes alone, is based on a comparison of existing CopsRef expressions with zero vs. one spatial relations. These expression groups differ not only in the presence of spatial relations but also in length, object classes, and attribute composition, so the observed accuracy differences cannot be attributed to spatial relations alone. A matched or controlled comparison (e.g., the same target entity with and without the spatial clause) is needed for this causal-sounding conclusion.","section":"§5.1 and RQ3, Tables 5 and 7"}],"minor_comments":[{"comment":"The Directional category contains only 29 single-relation expressions; the claim that all models 'struggle' with directional relations would be more convincing with confidence intervals over expressions or with a larger sample.","section":"§5.2, Table 2"},{"comment":"Precision and recall for the negation metrics are not defined; the text should specify the unit of analysis (e.g., per negation token, per expression, per predicted box) and how partial credit is assigned.","section":"§5.5, Table 10"},{"comment":"The caption 'Frequency of occurrence of relations' should say 'referring expressions'.","section":"Table 3"},{"comment":"The Vicuna reference is a blog post; the authors should cite the model's primary source or an official release.","section":"References"},{"comment":"The column header 'Coeff, 2-tailed test / Z-score' is confusing; separate columns or a note would help.","section":"Table 6"},{"comment":"The figure caption describes box colors for MGA-Net, Grounding DINO, and LLaVA, but not for OWL-ViT; if OWL-ViT is not shown, this should be stated.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a transparent and potentially useful empirical study, but the negation and visual-complexity analyses currently undermine two of the five headline findings. I recommend major revision rather than rejection: the underlying measurements are reproducible and the issues are fixable with additional analysis and re-reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper in one breath: it makes a genuine case for using Referring Expression Comprehension instead of caption/VQA to study spatial grounding in VLMs, and its category-wise accuracy tables are useful. But the two most advertised sub-analyses — negation and visual complexity — have methodological problems that undercut those specific claims. The per-category and spatial-composition results are the real contribution.\n\nWhat is new: the paper applies REC to the CopsRef dataset, with long expressions containing multiple spatial relations, and analyzes performance across 51 relations in 8 semantic categories. That is a legitimate move beyond prior work that used simplified expressions or synthetic data. The finding that spatial relations, when added to attributes, help grounding in an ambiguous setting is well supported by Tables 5 and 7. The finding that increasing the number of spatial relations hurts VLMs but not MGA-Net is also defensible and interesting. The qualitative examples in Figure 1 illustrate real failure modes.\n\nNow the soft spots. Section 5.5's negation analysis is the biggest issue. The paper restricts to expressions for which all three models already have IoU < 0.5, then reports how often each model 'fails to recognize at least 1 negation' within that subset. That cannot estimate the prevalence of negation failure, nor compare models, because the subset is defined by universal failure. The denominators — total negated expressions and matched non-negated controls — are missing. This does not support headline finding (5) as written.\n\nThe visual complexity split is less fatal but still problematic. Grounding DINO labels whether images contain single or multiple instances of the mentioned objects, and then Grounding DINO is itself evaluated on that split. Its multi-instance accuracy is therefore partly self-referential, and the other models inherit whatever detection bias the labeler has. An independent instance counter or human annotation would fix this.\n\nThere is also a statistical inconsistency: the reported Kendall tau z-scores do not match the stated formula. For example, tau = 0.73 with n = 12 gives z ≈ 3.30, not 2.05 as reported. Either the formula or the numbers are wrong, and the significance conclusion in Table 6 needs correction. Additionally, no code or data splits are released, so the relation-count and negation splits are not reproducible.\n\nWho is this for: people working on VLM spatial reasoning evaluation, and anyone building diagnostic benchmarks for grounding. The paper deserves a serious referee, but the referee should require re-analysis of the negation split with proper denominators, independent labeling for the visual complexity split, and a corrected statistical test. For my own work, I would cite the relation-count results if the splits are released; otherwise I would hesitate.","headline":"A solid empirical core on spatial relations in REC, undermined by a selection-biased negation analysis and a self-referential visual-complexity split; worth refereeing if those are fixed.","tokens_in":16273,"tokens_out":2772,"would_cite":true,"duration_ms":28312,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that Referring Expression Comprehension (REC), where a model must draw a bounding box around the object an expression names, is a sharper testbed than captioning or VQA for studying spatial reasoning because errors can…","keywords":["spatial reasoning","referring expression comprehension","vision-language models","grounding","negation","compositional reasoning","CopsRef","spatial relations"],"falsifier":"Re-run Table 8 with the single/multi-instance label assigned by human annotators (or by a second detector such as Faster R-CNN or a segmentation model) rather than by Grounding DINO, and check whether the 5.4-point average accuracy gap survives; a second check is to recompute the negation failure counts using a hand-verified gold set of negation-attribution labels instead of the IoU<0.5 criterion, since that criterion conflates box mislocalization with negation misinterpretation.","tokens_in":15324,"feed_emoji":"🧠","tokens_out":7274,"duration_ms":60536,"temperature":0.7,"pith_summary":"This paper argues that the Referring Expression Comprehension (REC) task—selecting the bounding box an expression points to—is a better microscope than captioning or VQA for studying spatial reasoning in vision-language models, because errors can be traced to specific words and relations. Using the CopsRef dataset, the authors compare a task-specific model (MGA-Net), two general VLMs (Grounding DINO and LLaVA), and a label-only detector baseline. Their central finding is that spatial relations are not uniformly hard: adding one spatial relation to object attributes usually improves grounding accuracy, but stacking two or three relations degrades the VLMs while leaving the compositionally trained MGA-Net roughly flat. Directional relations (e.g., 'around', 'through') and negated expressions ('not') are the two persistent trouble spots across all models. The paper also shows that images containing multiple instances of a named object lower accuracy for every model, and offers a category-by-category breakdown to guide future spatial-reasoning benchmarks.","feed_headline":"Spatial tests show why VLMs stumble on multi-step expressions","feed_subtitle":"Three-model CopsRef study: spatial relations help grounding; complexity, negation, and ambiguity expose model weaknesses.","key_machinery":"The load-bearing mechanism is the evaluation design: the CopsRef test set re-split along four interpretive axes—fine-grained spatial categories (8 categories covering 51 relations), spatial complexity (0–3 relations, distinguishing chained, conjunctive 'and', and disjunctive 'or' clauses), visual complexity (single vs multiple instances of a named object, determined by an automatic entity extractor plus an object-detector count), and negation presence (one or two 'not's). Comprehension accuracy is measured by IoU > 0.5 against the ground-truth box. The comparison across MGA-Net, Grounding DINO, LLaVA, and an OWL-ViT label-only baseline is what lets the paper attribute accuracy differences to architecture (compositional versus end-to-end, grounded versus instruction-tuned) rather than to dataset difficulty alone.","core_discovery":"On the paper's own terms, the discovery is that spatial grounding behavior in REC is governed by a small set of identifiable factors—spatial-relation category, number and composition of spatial clauses, visual multiplicity, and negation—and that model architecture predicts which factors are handled well. MGA-Net's graph-based compositional reasoning keeps accuracy stable as expressions grow from one to three spatial relations and excels at geometric projective relations such as 'left of'; Grounding DINO's grounded pre-training and LLaVA's full-image context give them an edge on vague proximity relations but make them more sensitive to expression length and negation. All models fail most on directional relations and on negations, with errors traceable qualitatively to partial interpretations of multi-clause expressions.","pith_inferences":["If the visual-complexity split is re-derived with an independent instance counter (human annotations or a different detector), the single- versus multi-instance accuracy gap in Table 8 could shrink or vanish; until that is checked, that particular claim rests on an oracle that is itself one of the evaluated models.","A testable extension is to add MGA-Net-style relation-graph decomposition to a LLaVA-class model and measure whether the multi-relation accuracy drop closes; the paper's architecture hypothesis predicts it would.","The negation results imply that simple data augmentation—adding negated spatial expressions to instruction-tuning mixtures—might close much of the gap without architectural change; this is directly suggested by MGA-Net's advantage when negatives are present in training.","Because CopsRef expressions are template-generated, the category-level accuracy pattern may partly reflect template artifacts rather than human spatial semantics; re-running the same splits on a human-written REC dataset like RefCOCOg would test the generality of the directional-and-negation weaknesses."],"forward_implications":["Adding a spatial relation to attribute-only descriptions improves grounding accuracy in ambiguous multi-object scenes, so attribute-only referring expressions understate what spatial language contributes.","Expressions with two or three spatial relations—especially chained or disjunctive ones—cut VLM accuracy by roughly 10–13 points, while MGA-Net's accuracy stays near 62–65%, showing that compositional architecture is a decisive variable.","Directional relations like 'around' and 'through' are the weakest category for every model, since their interpretation depends on the object configuration in each image.","Negated spatial expressions are a major blind spot: across 109 failed items, all models mis-handled 24 of 36 single-negation and 59 of 73 double-negation expressions, with Grounding DINO worst and MGA-Net better only when negation appeared in training.","Multi-instance images lower accuracy by about 5 percentage points on average, and the drop is largest for LLaVA, consistent with the claim that grounded pre-training matters for disambiguating among identical objects."],"supporting_citations":[{"why":"Supplies the CopsRef dataset, whose complex expressions and 17.4 objects per image make the REC analysis possible.","marker":"[Chen et al., 2020]"},{"why":"Defines MGA-Net, the task-specific REC model whose compositional graph attention is the paper's architecture counterweight to the VLMs.","marker":"[Zheng et al., 2020]"},{"why":"Grounding DINO provides the open-set detection results that set the VLM accuracy bar and that perform the instance counting for the visual complexity split.","marker":"[Liu et al., 2023b]"},{"why":"LLaVA is the general-purpose instruction-tuned VLM whose multi-relation and negation failures anchor the comparative claims.","marker":"[Liu et al., 2024]"},{"why":"OWL-ViT serves as the label-only object-detector baseline that isolates the contribution of spatial language.","marker":"[Minderer et al., 2022]"},{"why":"Supplies the eight-category spatial semantics taxonomy the paper adapts for its fine-grained relation split.","marker":"[Marchi Fagundes et al., 2021]"},{"why":"Establishes the IoU > 0.5 comprehension criterion used for all accuracy numbers.","marker":"[Yu et al., 2018]"},{"why":"Together with Yu et al., establishes the IoU > 0.5 comprehension criterion used throughout the evaluation.","marker":"[Liu et al., 2019]"}],"fun_headline_variants":["Why VLMs fall for negation and directional spatial terms","Graph reasoning keeps grounding stable as spatial expressions grow","Negation and multi-step relations trip up VLMs in spatial grounding","Architecture matters: which VLMs handle spatial complexity and negation","Spatial grounding blindspots: negation and directional cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The visual-complexity split assumes that the automatic entity extractor finds every object named in the expression and that the object detector (which is itself one of the evaluated models) reliably counts all instances of those entities; if either counts wrong, the single-versus-multi-instance accuracy comparison is confounded.","fun_headline_variants_meta":{"raw":{"variants":["Why VLMs fall for negation and directional spatial terms","Graph reasoning keeps grounding stable as spatial expressions grow","Negation and multi-step relations trip up VLMs in spatial grounding","Architecture matters: which VLMs handle spatial complexity and negation","Spatial grounding blindspots: negation and directional cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00222,"raw_usage":{"total_tokens":8547,"prompt_tokens":862,"completion_tokens":7685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":7604}},"tokens_in":478,"tokens_out":7685,"duration_ms":47733,"temperature":1.0,"reasoning_tokens":7604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:13:21.631102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Table 8 with the single/multi-instance label assigned by human annotators (or by a second detector such as Faster R-CNN or a segmentation model) rather than by Grounding DINO, and check whether the 5.4-point average accuracy gap survives; a second check is to recompute the negation failure counts using a hand-verified gold set of negation-attribution labels instead of the IoU<0.5 criterion, since that criterion conflates box mislocalization with negation misinterpretation.","supporting_citations":[],"review_version":1}