What is needed for simple spatial language capabilities in VQA?

Alexander Kuhnle; Ann Copestake

REVIEW 3 major objections 5 minor 13 references

What is needed for simple spatial language capabilities in VQA?

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Simple spatial language in VQA is powered by coordinate maps or FiLM-style early fusion.

desk verdict The coordinate-map branch of the central claim is overstated—coordinates help on implicit spatial comparisons but not on explicit relational statements where no model exceeds 80%—while the FiLM-style fusion result is solid and the ablations are genuinely informative. read the letter →

arxiv 1908.06336 v2 pith:M4HP7DQN submitted 2019-08-17 cs.CV cs.AIcs.CLcs.LG

Alexander Kuhnle , Ann Copestake This is my paper · ORCID

classification cs.CVcs.AIcs.CLcs.LG

keywords visualquestionansweringspatialrelationsfeature-wiselinearmodulationcoordinatemapsdiagnosticbenchmarksmodelablationsCLEVRsyntheticscenes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which architectural ingredient lets a visual question answering (VQA) model understand simple spatial relations like 'left of' and 'above'. Testing four recent models on diagnostic data restricted to spatial statements, it finds that two alternative techniques explain nearly all of the performance gap: attaching relative spatial coordinate maps to the image features, or fusing language and vision early via feature-wise linear modulation (FiLM) in convolutional layers. Stacked attention layers and relation modules contribute little on their own. The result matters because it isolates the load-bearing mechanism behind apparent differences among CLEVR-era models, and shows a plain CNN-LSTM can be boosted to near-FiLM accuracy with the right modification.

What carries the argument

The two load-bearing mechanisms are the coordinate map and feature-wise linear modulation (FiLM). The coordinate map is a fixed two-channel tensor giving each position of the $8 \times 8$ image feature grid its relative $(x,y)$ coordinates, concatenated to the features at the start of the model's core module. FiLM lets language features produce per-channel scale and shift parameters that modulate the visual feature map inside convolutional layers; with four stacked $3 \times 3$ convolutional layers the whole feature grid is covered and local relative positions can be composed into global spatial relations. The paper's evidence comes from targeted ablations—swapping coordinates on and off, replacing fully-connected mixing with convolutions, and replacing concatenation with FiLM fusion—which jointly show that either mechanism alone reproduces the top performance.

What would settle it

Retrain the CNN-LSTM baseline for 200k iterations and re-run the ablation; if the extended baseline reduces the gap to FiLM without coordinates or early fusion, the attribution of the gains to those two techniques would be weakened. Alternatively, apply the 'FiLM + convolutions' modification to a spatial dataset with non-uniform object scales, where a single fixed $8 \times 8$ coordinate map is not translation-invariant.

Watch

Extended reading notes

Core claim

The central claim is that high-level modules—SAN's stacked attention, RelNet's relation module, MC's multimodal core—are not what gives a model spatial language competence. Instead, one of two low-level techniques is responsible: concatenating a fixed map of relative spatial coordinates to the image feature grid, or modulating visual features by language-derived scale and shift parameters (FiLM) inside stacked convolutional layers. The paper shows this by ablating all four models: adding coordinates to MC, SAN, and even the CNN-LSTM baseline lifts them to roughly RelNet's level; removing coordinates from FiLM does not hurt it; and rewiring MC or CNN-LSTM to use FiLM fusion with convolutions (instead of concatenation with fully-connected layers) brings them to approximately FiLM's accuracy. RelNet cannot take the convolution route because its pairwise combinations destroy the two-dimensional arrangement of image features, and SAN's stacked attention does not integrate the changes.

Load-bearing premise

The central comparison rests on the CNN-LSTM baseline being trained for only 100k iterations while every other model gets 200k, and on the assumption that keeping hyperparameters uniform across models does not disadvantage any one architecture.

Editorial extensions

If this is right

On explicit spatial statements, FiLM reaches about 77% accuracy; on comparative and superlative forms it and RelNet reach about 97%, while SAN stays near or slightly above the CNN-LSTM baseline.
Adding relative coordinate maps lifts MC, SAN, and the CNN-LSTM baseline to around the level of RelNet, showing the coordinate trick transfers across architectures.
Replacing the late-fusion CNN-LSTM's core with early FiLM fusion and convolutional layers boosts it to nearly FiLM's accuracy, so the advantage is not tied to the full FiLM architecture.
The relation module of RelNet does not, by itself, improve spatial performance: MC with coordinates matches RelNet with coordinates despite lacking pairwise relation processing.
Four $3 \times 3$ convolutional FiLM layers span the entire $8 \times 8$ feature map, so the model can build global spatial judgments from local positional evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the pattern generalizes beyond synthetic shapes, then comparisons of VQA models on spatial questions should control for whether coordinate maps or FiLM-style fusion are present before crediting attention or relation mechanisms.
A testable extension: on datasets where spatial terms depend on object orientation or reference frames, fixed coordinate maps may be insufficient and the FiLM-plus-convolutions route may need additional geometric priors.
The near-10% gain that coordinates give even to the late-fusion baseline suggests that positional signal can survive average pooling, which could inform lightweight systems that cannot afford full FiLM conditioning.
One should not assume that the two techniques are equivalently robust outside this 64x64 grid setting; rescaling images or using variable-resolution feature maps would be a natural stress test for both mechanisms.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline claim overshoots the data. Saying that two techniques 'enable high accuracy for VQA instances involving spatial relations' is too strong when on SPATIAL-EXPLICIT, the most direct test of spatial relations, no model gets above 80%, and coordinate concatenation alone does not even help SAN there. The coordinate result really holds on the implicit comparative/superlative datasets. The other branch—early FiLM-style modulation plus convolutions—is much better supported, and the paper correctly identifies that as the more effective method. So the central dichotomy needs qualification, not abandonment.

What the paper does well: it is a clean, controlled diagnostic comparison of SAN, RelNet, FiLM, and MC on ShapeWorld spatial data. The within-model modifications are the right way to attribute performance—adding/removing coordinate maps, swapping fusion and convolution layers—and the three-run averages with min/max shading are clearly reported. The finding that stacked attention and the relation module contribute little for this task, while coordinates and FiLM-style early fusion do, is a useful practical result. The ablations that transfer the FiLM trick to MC and the CNN-LSTM baseline are the strongest part of the paper.

Soft spots worth naming: first, the coordinate overgeneralization mentioned above. Second, the CNN-LSTM baseline trains for 100k iterations versus 200k for every other model; that is a real asymmetry and could make the baseline look weaker than it is. Third, hyperparameters are not perfectly matched across models—RelNet uses a 128-dim LSTM, SAN uses 256-dim attention layers—so exact cross-model comparisons are slightly confounded. These are moderate caveats, not fatal ones. The core attribution story survives.

This paper is for researchers working on VQA or visual reasoning diagnostics who want to know which architectural pieces actually buy spatial language competence. It is not a landmark, but it is honest and reproducible, with code and public data. I would cite it for the specific finding that relation modules and stacked attention do not drive spatial performance here. If I were editing, I would send it to peer review; the authors should soften the coordinate claim and address the training schedule asymmetry, but the work deserves referee time.

Referee Report

3 major / 5 minor

Summary. The paper asks which architectural ingredients are necessary for VQA models to handle simple spatial language. Using diagnostic data generated with ShapeWorld, it compares CNN-LSTM, SAN, RelNet, FiLM, and MC on three caption types (SPATIAL-EXPLICIT, SPATIAL-COMPARATIVE, SPATIAL-SUPERLATIVE). The authors then perform targeted modifications: adding or removing coordinate maps, replacing fully-connected layers with convolutions, and changing fusion from concatenation to FiLM-style modulation. They conclude that two techniques suffice for high accuracy on spatial-relation VQA: concatenating image features with relative spatial coordinates, and early fusion via feature-wise linear modulation combined with convolutional layers. They also argue that stacked attention and the RelNet relation module do not independently contribute.

Significance. If the claims are appropriately scoped, this is a useful diagnostic result for VQA architecture design: it isolates which components matter for a specific capability and would suggest that simpler models augmented with coordinate maps or FiLM+convolutions can replace more complex attention/relation machinery for this kind of spatial reasoning. The study is well designed in several respects: it uses controlled diagnostic data, runs each experiment three times with min/max shading, performs targeted ablations, and releases code. The main limitation is that the coordinate-map conclusion is not supported on the explicit spatial-relation dataset, and the comparison has some hyperparameter and training-schedule imbalances that make the attribution of gains to specific components less secure.

major comments (3)

[§1 and §5] The abstract and introduction claim that concatenating relative spatial coordinates is one of two techniques that enable models to achieve high accuracy for VQA instances involving spatial relations. This claim is not supported on SPATIAL-EXPLICIT: §5 reports that no model exceeds 80% accuracy there, and Figure 3 shows that adding coordinates to SAN does not improve performance on this dataset. The coordinate evidence is strong only for SPATIAL-COMPARATIVE and SPATIAL-SUPERLATIVE. Please qualify the claim to implicit comparative/superlative relations, or provide additional evidence that coordinate maps help on explicit relational statements.
[§4] The CNN-LSTM baseline is trained for 100k iterations while all other models are trained for 200k. Since much of the paper's argument is framed as improvement over the CNN-LSTM baseline, this asymmetry could overstate the improvement if the baseline would continue to learn with more training. Please retrain the baseline for 200k iterations, or provide learning curves demonstrating that the baseline has saturated by 100k.
[§3] The text says generic parts are kept the same 'to enable a fair comparison', but the language module sizes differ: RelNet uses a 128-dimensional LSTM, FiLM a 512-dimensional GRU, and SAN uses 256-dimensional attention layers. These differences confound the attribution of performance gains to the core module under study. Please either match these hyperparameters across models or provide a sensitivity analysis showing that the conclusions are robust to these choices.

minor comments (5)

[§2] The notation 'six/eight spatial relations' is unclear; please state explicitly which relations are available for each dataset and why 'behind' and 'in front of' are excluded from the implicit variants.
[§3] The sentence 'processed by an LSTM, or GRU in case of FiLM, of size 512, or 128 in case of RelNet' is ambiguous; please restructure to make the size for each model explicit.
[Figures 2 and 3] The +/–/* markers in Figure 3 and the ablation variants in Figures 4–6 are explained only in the captions; consider adding a legend or defining the markers directly in the figure for readability.
[Throughout] Dataset names such as 'SPATIAL -EXPLICIT' contain inconsistent spacing; please unify to a single style, e.g., 'SPATIAL-EXPLICIT'.
[References] Reference [6] is an arXiv e-print; if a peer-reviewed version of ShapeWorld exists, please cite that instead or in addition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's conclusions rest on direct model comparisons and ablations, not on fitted inputs or self-cited constraints.

full rationale

This paper is purely empirical and contains no derivation chain in which a prediction is recovered from its own inputs. The central claim that coordinate concatenation or FiLM-style early fusion with convolutions drives spatial-language performance is supported by controlled comparisons and targeted ablations: the authors add coordinate maps to models that lack them, remove them from models that have them, replace fully-connected layers with convolutions, and switch concatenation to FiLM fusion. These manipulations are independent of the conclusion and are not fitted to a subset of the test data. The only self-citation is the ShapeWorld simulator [6], which is used as a fixed data generator, not tuned to model outputs and not defined in terms of the target results, so it is not load-bearing in a circular sense. The unequal training schedules (100k iterations for the CNN-LSTM baseline versus 200k for other models) and uniform hyperparameter choices are potential threats to fairness, but they are correctness risks rather than circularity. Therefore no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and does not introduce new mathematical axioms, fitted constants, or invented entities. It depends on the validity of the ShapeWorld data distribution as a proxy for spatial language, on faithful model implementations, and on the fairness of the training schedule. No numbers are fitted to make the result work.

assumptions (3)

domain assumption ShapeWorld's generation process produces a representative distribution of simple spatial language statements for evaluating spatial capabilities.
The paper uses ShapeWorld (a simulator authored by the authors) to generate all evaluation data. The conclusions about 'what is needed' generalize only if these generated statements reflect the spatial language phenomena of interest. This assumption enters in Section 2.
domain assumption The implementation of each model faithfully represents the architectural contribution of the original paper, and the hyperparameter choices made for uniformity do not change the relative conclusions.
In Section 3, the authors state they keep generic parts consistent and identify each model's core module. The fairness of the comparison depends on this assumption, especially because some hyperparameters differ (e.g., RelNet's 128-dim language embedding, SAN's 256-dim layers).
domain assumption The CNN-LSTM baseline's performance after 100k iterations is comparable to what it would achieve with 200k iterations, making it a fair baseline.
Section 4 states models train for 200k iterations but the baseline only for 100k. If the baseline would improve with more training, relative gains would be overstated. The paper does not justify this choice.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What is needed for simple spatial language capabilities in VQA?." pith.science (2026). https://pith.science/paper/M4HP7DQN

@misc{pith2026190806336,
  author       = {Pith},
  title        = {Pith review of: What is needed for simple spatial language capabilities in VQA?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4HP7DQN}},
  note         = {Machine review of arXiv:1908.06336}
}

read the original abstract

Visual question answering (VQA) comprises a variety of language capabilities. The diagnostic benchmark dataset CLEVR has fueled progress by helping to better assess and distinguish models in basic abilities like counting, comparing and spatial reasoning in vitro. Following this approach, we focus on spatial language capabilities and investigate the question: what are the key ingredients to handle simple visual-spatial relations? We look at the SAN, RelNet, FiLM and MC models and evaluate their learning behavior on diagnostic data which is solely focused on spatial relations. Via comparative analysis and targeted model modification we identify what really is required to substantially improve upon the CNN-LSTM baseline.

Figures

Figures reproduced from arXiv: 1908.06336 by the authors.

**Figure 1.** An example image plus three valid captions containing simple spatial relations. Each one illustrates a different way of how such a relation may be realized in language. The statements in our experiments may also be wrong, and thus basically act as yes/no questions, which the model has to answer correctly, that is, infer whether or not a statement agrees with an image. image-caption agreement, but the data can be tr… view at source ↗

**Figure 2.** Performance curves over the course of training (x-axis: iterations in 1000, y-axis: accuracy). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Performance curves, for models with (+ ) or without (– ) coordinate map, or with early FiLM fusion and convolutions instead of concatenation and fully-connected layers (*). the image features. This minor detail infuses useful spatial information and thus relieves the core module of having to learn the concept of relative spatial position from scratch. Since the modification can easily be applied to the other models … view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Accuracy performance curves over the course of training, for the FiLM model and various [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Accuracy performance curves over the course of training, for the MC model and various [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Accuracy performance curves over the course of training, for the CNN-LSTM model and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

[1]

Lawrence Zitnick, and Devi Parikh

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, ICCV 2015, pages 2425–2433, 2015

work page 2015
[2]

Learning to reason: End-to-end module networks for visual question answering

Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision , ICCV 2017, pages 804–813, 2017

work page 2017
[3]

Hudson and Christopher D

Drew A. Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. In Proceedings of the International Conference on Learning Representations , ICLR 2018, 2018

work page 2018
[4]

Lawrence Zitnick, and Ross Girshick

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 1988–1997, 2017

work page 2017
[5]

Inferring and executing programs for visual rea- soning

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual rea- soning. In Proceedings of the IEEE International Conference on Computer Vision , ICCV 2017, pages 3008–3017, 2017

work page 2017
[6]

ShapeWorld - a new test methodology for multimodal language understanding

Alexander Kuhnle and Ann Copestake. ShapeWorld - a new test methodology for multimodal language understanding. ArXiv e-prints 1704.04517, 2017

arXiv 2017
[7]

An intriguing failing of convolutional neural networks and the CoordConv solution

Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the CoordConv solution. In Advances in Neural Information Processing Systems 31 , pages 9605–9616, 2018

work page 2018
[8]

The visual QA devil in the details: The impact of early fusion and batch norm on CLEVR

Mateusz Malinowski and Carl Doersch. The visual QA devil in the details: The impact of early fusion and batch norm on CLEVR. In Proceedings of the European Conference on Computer Vision, ECCV 2018, 2018

work page 2018

Show all 13 references

[9]

Transparency by design: Closing the gap between performance and interpretability in visual reasoning

David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , CVPR 2018, pages 4942–4950, 2018

2018
[10]

Courville

Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI, 2018

2018
[11]

Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy P. Lillicrap. A simple neural network module for relational reasoning. In Proceedings of the Annual Conference on Neural Information Processing Systems , pages 496...

2017
[12]

A dataset and architecture for visual reasoning with a working memory

Robert Guangyu Yang, Igor Ganichev, Xiao Jing Wang, Jonathon Shlens, and David Sussillo. A dataset and architecture for visual reasoning with a working memory. In Proceedings of the European Conference on Computer Vision, ECCV 2018, pages 729–745, 2018

2018
[13]

Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 21–29, 2016. 5 Supplementary material to What is...

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Lawrence Zitnick, and Devi Parikh

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, ICCV 2015, pages 2425–2433, 2015

work page 2015

[2] [2]

Learning to reason: End-to-end module networks for visual question answering

Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision , ICCV 2017, pages 804–813, 2017

work page 2017

[3] [3]

Hudson and Christopher D

Drew A. Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. In Proceedings of the International Conference on Learning Representations , ICLR 2018, 2018

work page 2018

[4] [4]

Lawrence Zitnick, and Ross Girshick

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pages 1988–1997, 2017

work page 2017

[5] [5]

Inferring and executing programs for visual rea- soning

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual rea- soning. In Proceedings of the IEEE International Conference on Computer Vision , ICCV 2017, pages 3008–3017, 2017

work page 2017

[6] [6]

ShapeWorld - a new test methodology for multimodal language understanding

Alexander Kuhnle and Ann Copestake. ShapeWorld - a new test methodology for multimodal language understanding. ArXiv e-prints 1704.04517, 2017

arXiv 2017

[7] [7]

An intriguing failing of convolutional neural networks and the CoordConv solution

Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the CoordConv solution. In Advances in Neural Information Processing Systems 31 , pages 9605–9616, 2018

work page 2018

[8] [8]

The visual QA devil in the details: The impact of early fusion and batch norm on CLEVR

Mateusz Malinowski and Carl Doersch. The visual QA devil in the details: The impact of early fusion and batch norm on CLEVR. In Proceedings of the European Conference on Computer Vision, ECCV 2018, 2018

work page 2018