Pith. sign in

REVIEW 4 major objections 6 minor 13 references

Exploring Spatial Language Grounding Through Referring Expressions

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read This paper argues that Referring Expression Comprehension (REC), where a model must draw a bounding box around the object an expression names, is a sharper testbed than captioning or VQA for studying spatial reasoning because errors can…

desk verdict A solid empirical core on spatial relations in REC, undermined by a selection-biased negation analysis and a self-referential visual-complexity split; worth refereeing if those are fixed. read the letter →

arxiv 2502.04359 v1 pith:UR5W6RKV submitted 2025-02-04 cs.CL cs.AIcs.CV

classification cs.CLcs.AIcs.CV
keywords spatialreasoningreferringexpressioncomprehensionvision-languagemodelsgroundingnegationcompositionalCopsRefrelations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the Referring Expression Comprehension (REC) task—selecting the bounding box an expression points to—is a better microscope than captioning or VQA for studying spatial reasoning in vision-language models, because errors can be traced to specific words and relations. Using the CopsRef dataset, the authors compare a task-specific model (MGA-Net), two general VLMs (Grounding DINO and LLaVA), and a label-only detector baseline. Their central finding is that spatial relations are not uniformly hard: adding one spatial relation to object attributes usually improves grounding accuracy, but stacking two or three relations degrades the VLMs while leaving the compositionally trained MGA-Net roughly flat. Directional relations (e.g., 'around', 'through') and negated expressions ('not') are the two persistent trouble spots across all models. The paper also shows that images containing multiple instances of a named object lower accuracy for every model, and offers a category-by-category breakdown to guide future spatial-reasoning benchmarks.

What carries the argument

The load-bearing mechanism is the evaluation design: the CopsRef test set re-split along four interpretive axes—fine-grained spatial categories (8 categories covering 51 relations), spatial complexity (0–3 relations, distinguishing chained, conjunctive 'and', and disjunctive 'or' clauses), visual complexity (single vs multiple instances of a named object, determined by an automatic entity extractor plus an object-detector count), and negation presence (one or two 'not's). Comprehension accuracy is measured by IoU > 0.5 against the ground-truth box. The comparison across MGA-Net, Grounding DINO, LLaVA, and an OWL-ViT label-only baseline is what lets the paper attribute accuracy differences to architecture (compositional versus end-to-end, grounded versus instruction-tuned) rather than to dataset difficulty alone.

What would settle it

Re-run Table 8 with the single/multi-instance label assigned by human annotators (or by a second detector such as Faster R-CNN or a segmentation model) rather than by Grounding DINO, and check whether the 5.4-point average accuracy gap survives; a second check is to recompute the negation failure counts using a hand-verified gold set of negation-attribution labels instead of the IoU<0.5 criterion, since that criterion conflates box mislocalization with negation misinterpretation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that spatial grounding behavior in REC is governed by a small set of identifiable factors—spatial-relation category, number and composition of spatial clauses, visual multiplicity, and negation—and that model architecture predicts which factors are handled well. MGA-Net's graph-based compositional reasoning keeps accuracy stable as expressions grow from one to three spatial relations and excels at geometric projective relations such as 'left of'; Grounding DINO's grounded pre-training and LLaVA's full-image context give them an edge on vague proximity relations but make them more sensitive to expression length and negation. All models fail most on directional relations and on negations, with errors traceable qualitatively to partial interpretations of multi-clause expressions.

Load-bearing premise

The visual-complexity split assumes that the automatic entity extractor finds every object named in the expression and that the object detector (which is itself one of the evaluated models) reliably counts all instances of those entities; if either counts wrong, the single-versus-multi-instance accuracy comparison is confounded.

Editorial extensions

If this is right

  • Adding a spatial relation to attribute-only descriptions improves grounding accuracy in ambiguous multi-object scenes, so attribute-only referring expressions understate what spatial language contributes.
  • Expressions with two or three spatial relations—especially chained or disjunctive ones—cut VLM accuracy by roughly 10–13 points, while MGA-Net's accuracy stays near 62–65%, showing that compositional architecture is a decisive variable.
  • Directional relations like 'around' and 'through' are the weakest category for every model, since their interpretation depends on the object configuration in each image.
  • Negated spatial expressions are a major blind spot: across 109 failed items, all models mis-handled 24 of 36 single-negation and 59 of 73 double-negation expressions, with Grounding DINO worst and MGA-Net better only when negation appeared in training.
  • Multi-instance images lower accuracy by about 5 percentage points on average, and the drop is largest for LLaVA, consistent with the claim that grounded pre-training matters for disambiguating among identical objects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the visual-complexity split is re-derived with an independent instance counter (human annotations or a different detector), the single- versus multi-instance accuracy gap in Table 8 could shrink or vanish; until that is checked, that particular claim rests on an oracle that is itself one of the evaluated models.
  • A testable extension is to add MGA-Net-style relation-graph decomposition to a LLaVA-class model and measure whether the multi-relation accuracy drop closes; the paper's architecture hypothesis predicts it would.
  • The negation results imply that simple data augmentation—adding negated spatial expressions to instruction-tuning mixtures—might close much of the gap without architectural change; this is directly suggested by MGA-Net's advantage when negatives are present in training.
  • Because CopsRef expressions are template-generated, the category-level accuracy pattern may partly reflect template artifacts rather than human spatial semantics; re-running the same splits on a human-written REC dataset like RefCOCOg would test the generality of the directional-and-negation weaknesses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes Referring Expression Comprehension (REC) as a platform for diagnosing spatial reasoning in vision-language models. Using the CopsRef test set, the authors categorize 51 spatial relations into 8 semantic classes, split expressions by number and type of spatial relations (none, one, two-chained, two-and, two-or, three), and evaluate MGA-Net, Grounding DINO, LLaVA, and an OWL-ViT label-only baseline. The headline findings are that spatial relations often improve grounding relative to attribute-only expressions; increasing relation count hurts zero-shot VLMs but not the task-specific MGA-Net; directional relations are hardest; task-specific models and VLMs differ on geometric versus proximity relations; and all models struggle with negated relations. The paper also reports a visual-complexity split and qualitative case studies.

Significance. If the analysis were sound, the paper would make a useful contribution as a fine-grained evaluation of spatial grounding along several dimensions not jointly covered by prior VQA/captioning work. The authors are careful to report multiple runs, disclose training constraints, make their evaluation splits explicit, and include an appendix documenting models that failed to produce usable outputs (InstructBLIP, OpenFlamingo). These are genuine strengths. The per-category accuracy measurements in Tables 4-7 are straightforward and likely reproducible. However, the two novel analysis axes most emphasized in the abstract and findings—negation handling and visual complexity—rest on methodological choices that undermine the corresponding conclusions. The central platform claim for negation (finding 5) is not supported by the present design; the visual-complexity result is partly self-referential for Grounding DINO; and the architecture-level interpretation of MGA-Net's stability is confounded with training on the evaluation set.

major comments (4)
  1. [§5.5, Table 9] The negation analysis is selection-biased. The authors first restrict to expressions with one or two 'not's for which all three models already have IoU < 0.5 (36 and 73 expressions), then report how often each model 'failed to recognize at least 1 negation' within that subset. Because every expression in the subset is already a failure for all models, the rows cannot estimate the prevalence of negation errors, the relative difficulty of one vs. two negations, or which model is better. In particular, the statement that 'Grounding DINO has the highest number of failure instances' and the conclusion that 'LLaVA handles negations better' are artifacts of conditioning on universal failure. To support finding (5), the paper needs denominators over all negated expressions in CopsRef and a matched non-negated control set (e.g., expressions with comparable length, entities, and attributes).
  2. [§4.2 and Table 8] The visual-complexity split is constructed by using Grounding DINO to count instances of entities extracted by spaCy, and the same model is then scored on that split. Any systematic over- or under-counting by Grounding DINO, or errors in spaCy entity extraction, directly contaminates the single- vs. multi-instance label and, for the Grounding DINO row, makes the comparison partly self-referential. The authors should validate the instance counts against human annotation or an independent detector, and either report results excluding the model used for splitting or analyze sensitivity to the label set.
  3. [§5.3, Table 7 (RQ4)] The claim that MGA-Net's compositional architecture explains its stable accuracy across zero, one, two, and three spatial relations is confounded: MGA-Net was trained on CopsRef (for ten epochs, after being initialized from a detector), whereas Grounding DINO, LLaVA, and OWL-ViT were evaluated zero-shot. Better performance on the training distribution is an alternative explanation that is not controlled for. At minimum, the paper should temper the architectural conclusion and discuss this confound in Section 5.3.
  4. [§5.1 and RQ3, Tables 5 and 7] Finding (1), that spatial relations improve grounding relative to attributes alone, is based on a comparison of existing CopsRef expressions with zero vs. one spatial relations. These expression groups differ not only in the presence of spatial relations but also in length, object classes, and attribute composition, so the observed accuracy differences cannot be attributed to spatial relations alone. A matched or controlled comparison (e.g., the same target entity with and without the spatial clause) is needed for this causal-sounding conclusion.
minor comments (6)
  1. [§5.2, Table 2] The Directional category contains only 29 single-relation expressions; the claim that all models 'struggle' with directional relations would be more convincing with confidence intervals over expressions or with a larger sample.
  2. [§5.5, Table 10] Precision and recall for the negation metrics are not defined; the text should specify the unit of analysis (e.g., per negation token, per expression, per predicted box) and how partial credit is assigned.
  3. [Table 3] The caption 'Frequency of occurrence of relations' should say 'referring expressions'.
  4. [References] The Vicuna reference is a blog post; the authors should cite the model's primary source or an official release.
  5. [Table 6] The column header 'Coeff, 2-tailed test / Z-score' is confusing; separate columns or a note would help.
  6. [Figure 1] The figure caption describes box colors for MGA-Net, Grounding DINO, and LLaVA, but not for OWL-ViT; if OWL-ViT is not shown, this should be stated.

Circularity Check

2 steps flagged · score 6.0 of 10

Negation finding is selected into existence: the §4.2 split keeps only expressions where all three models already fail, and §5.5 then reports that all models struggle with negation; the visual-complexity split is labeled by the very Grounding DINO model that is scored on it.

  1. self definitional [Section 4.2 (Negation Analysis Split) and Section 5.5 (Impact of Negation)]
    "Then, we collected those expressions for which all three models give an IoU of less than 0.5. ... Hence, to answer RQ5, we observe that while all REC models face issues with recognizing negations, some models are comparatively better at handling them."

    The negation-analysis subset is defined by the very property the section concludes: every included expression is one on which all three models already fail (IoU<0.5). The reported 'Total failure' counts in Table 9 and the statement that 'all REC models face issues with recognizing negations' are therefore restatements of the inclusion rule rather than measurements. Because the table provides no denominator of all negated expressions in CopsRef and no matched non-negated control set, the comparative claims ('Grounding DINO has the highest number of failure instances', 'LLaVA handles negations better') are conditioned on a set chosen for universal failure and cannot support RQ5 as stated.

  2. self definitional [Section 4.2 (Visual Complexity Split) and Table 8]
    "We perform this splitting by first collecting the entities in each expression using spaCy and then employing Grounding DINO to find the number of instances in the image for each of the collected entities."

    Grounding DINO is used to produce the instance-count labels that define the single- vs multi-instance split, and then the same model is scored on that split in Table 8. Any detection error by Grounding DINO changes the label: a missed instance makes an image 'single' and lets the model appear to succeed at single-instance grounding, while a spurious detection can add a 'multi' example it then fails. The multi-instance accuracy of Grounding DINO is thus partly a function of its own detections, so the split is not an independent test condition for that model; the other models inherit the same model-dependent labels.

full rationale

Most of the paper is an empirical benchmark comparison with no fitted parameters or derived predictions, so the bulk of the results are independent measurements against the fixed CopsRef test set, and there are no load-bearing self-citations. The circularity is concentrated in two constructed splits. First, the negation analysis selects only expressions on which every model already scores IoU<0.5; from this subset the paper concludes that all models struggle with negation and compares their negation-failure counts. That conclusion is a restatement of the selection rule, and the cross-model comparison lacks the denominators needed to estimate prevalence or relative difficulty. Second, the visual-complexity split is generated by Grounding DINO's own instance counts and then used to score Grounding DINO, making its single- vs multi-instance accuracy partially self-referential. Neither issue invalidates the other findings, but the negation analysis underpins one of the five headline claims, so the circularity is partial rather than total.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper's central claims rest on dataset representativeness, the validity of the split-construction procedures, and comparability across differently-trained models; no free parameters or invented entities are introduced.

assumptions (6)
  • domain assumption The CopsRef test set is a representative sample of complex real-world referring expressions for evaluating spatial grounding.
    The paper selects CopsRef over RefCOCO/RefCOCOg/CLEVR-Ref+ based on expression length and object counts (Section 3); the generalizability of all findings is bounded by this dataset choice.
  • domain assumption IoU > 0.5 is a valid correctness criterion for grounding.
    Adopted from prior REC works (Yu et al., 2018; Liu et al., 2019) in Section 5; changes to the threshold would change accuracies and possibly rankings.
  • ad hoc to paper Grounding DINO can accurately identify all instances of every entity mentioned in an expression, and spaCy extracts the correct entity set, for constructing the visual complexity split.
    Section 4.2 Visual Complexity Split uses spaCy and Grounding DINO to label images as single- or multi-instance; the evaluated model itself contributes to the label it is later scored on.
  • ad hoc to paper For the negation analysis, selecting expressions where all three models already produced IoU < 0.5 still permits conclusions about relative negation-handling ability.
    Section 4.2 Negation Analysis Split and Section 5.5 build the analysis exclusively on cases where all models failed; the sample is conditioned on the outcome being studied.
  • domain assumption Category rankings computed from Table 5 are comparable despite MGA-Net being re-trained for 10 epochs on a partial ResNet-101 backbone while the VLMs are evaluated zero-shot.
    Section 4.1/5.1 acknowledge the training discrepancy and restrict claims to relative per-category performance; if MGA-Net's training is not representative, architecture-level conclusions (RQ2, RQ4) could change.
  • domain assumption The spatial relation taxonomy of Marchi Fagundes et al. (2021), with 'Cardinal Direction' replaced by 'Absolute', maps cleanly onto CopsRef expressions.
    Section 3 and Appendix A.1; some relations are 'Unallocated', and the Orientation category has no single-relation expressions, so category coverage is incomplete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Spatial Language Grounding Through Referring Expressions." pith.science (2026). https://pith.science/paper/UR5W6RKV

@misc{pith2026250204359,
  author       = {Pith},
  title        = {Pith review of: Exploring Spatial Language Grounding Through Referring Expressions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UR5W6RKV}},
  note         = {Machine review of arXiv:2502.04359}
}
read the original abstract

Spatial Reasoning is an important component of human cognition and is an area in which the latest Vision-language models (VLMs) show signs of difficulty. The current analysis works use image captioning tasks and visual question answering. In this work, we propose using the Referring Expression Comprehension task instead as a platform for the evaluation of spatial reasoning by VLMs. This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not'). In our analysis, we use task-specific architectures as well as large VLMs and highlight their strengths and weaknesses in dealing with these specific situations. While all these models face challenges with the task at hand, the relative behaviors depend on the underlying models and the specific categories of spatial semantics (topological, directional, proximal, etc.). Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.

Figures

Figures reproduced from arXiv: 2502.04359 by the authors.

Figure 1
Figure 1. Figures for Qualitative analysis. In the figures, the green box is the ground-truth bounding box. The red, blue, and yellow boxes are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

  1. [1]

    Eg:- man on the right that is standing and wearing gray pant

    Absolute: Consists of relations that describe the loca- tion of an object in an absolute manner and not in rela- tion to another object. Eg:- man on the right that is standing and wearing gray pant

  2. [2]

    Eg:- The large poster that is leaning against the wall

    Adjacency: Consists of relations that describe the phys- ical proximity of two objects. Eg:- The large poster that is leaning against the wall

  3. [3]

    The interpretation of these relations heavily relies on the configuration of the involved ob- jects and/or the dynamic spatial relationship between them

    Directional: Consists of dynamic action verbs / direc- tional relations. The interpretation of these relations heavily relies on the configuration of the involved ob- jects and/or the dynamic spatial relationship between them. Eg:- The gray car that is driving down the road

  4. [4]

    Eg:- The sitting dog that is facing the window that is to the right of the mirror

    Orientation: Consists of relations which describe the orientation of an object w.r.t another object. Eg:- The sitting dog that is facing the window that is to the right of the mirror

  5. [5]

    Eg:- The black oven that is above the drawer

    Projective: Consists of relations that indicate the con- crete spatial relationship between two objects, i.e., these relations can be quantified in terms of the coordinates of the two objects. Eg:- The black oven that is above the drawer

  6. [6]

    Eg:- The blue chair that is close to the white monitor

    Proximity: Consists of relations that indicate that two objects are near each other without giving a specific di- rectional relationship. Eg:- The blue chair that is close to the white monitor

  7. [7]

    Topological: Consists of relations that indicate the broader arrangement or the containment of an object w.r.t another object Eg:- The silver train that is at the colorful station

  8. [8]

    A.2 Other Models In our analysis, we also experimented with InstructBLIP[Dai et al., 2023] and OpenFlamingo [Awadalla et al., 2023] mod- els

    Unallocated: Consists of relations that cannot be allo- cated to any of the above categories. A.2 Other Models In our analysis, we also experimented with InstructBLIP[Dai et al., 2023] and OpenFlamingo [Awadalla et al., 2023] mod- els. These models are general-purpose VLMs with Instruct- BLIP working in the zero-shot model and OpenFlamingo in the few-shot...

Show all 13 references
  1. [11]

    Bounding Boxes: bounding box list; Referring Expres- sion: Refexp; The index of the output bounding box is:

  2. [12]

    Bounding Boxes: bounding box list; Referring Expres- sion: Refexp; The coordinates of the output bounding box are:

  3. [13]

    But for the third prompt, the model has no access to pre-detected candidate bounding boxes in the im- age

    Provide the bounding box coordinates for: ”Refexp” In both prompts, the ‘bounding box list’ consists of the co- ordinates of the detected bounding boxes in the image being passed as the input, along with indices for each bounding box, starting from ‘1’. But for the third promp...

  4. [166]

    [Gokhale et al., 2022] Tejas Gokhale, Hamid Palangi, Be- smira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang

    Springer, 2025. [Gokhale et al., 2022] Tejas Gokhale, Hamid Palangi, Be- smira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking Spatial Relationships in Text-to-Image Generation. arXiv preprint arXiv:2212.10015, 2022. [Hsu et al., 2024...

  5. [755]

    [Mirzaee et al., 2021] Roshanak Mirzaee, Hossein Ra- jaby Faghihi, Qiang Ning, and Parisa Kordjmashidi

    Springer, 2022. [Mirzaee et al., 2021] Roshanak Mirzaee, Hossein Ra- jaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. arXiv preprint arXiv:2104.05832 , 2021. [Murty et al., 2023] Shikhar Murty, Pratyusha ...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.