{"id":"ba453213-d092-4428-b97d-ba69cffbb9a3","arxiv_id":"1908.06336","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding relative spatial coordinate maps or using early FiLM-style modulation with convolutions, rather than attention or relation modules, drives performance on simple spatial language in VQA.","lead":"This paper tests which parts of visual question answering models actually matter for understanding simple spatial words like 'left of' and 'above'. It finds that adding relative coordinate maps, or using early feature-wise modulation with convolutions, helps most, while stacked attention and relation modules do not.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coordinate-map branch of the central claim is not supported on the explicit spatial-relation dataset, where no model exceeds 80%.","rationale":"The reader's stated weakest assumption was the 100k-versus-200k training schedule for the CNN-LSTM baseline. I agree that this is worth checking, but it is not the most load-bearing issue: even if the baseline were trained for 200k iterations, it would not rescue the coordinate branch on SPATIAL-EXPLICIT, where Figure 3 already shows coordinate addition failing for SAN and no high-accuracy coordinate-only result is reported for the explicit case. The more central problem is that the paper's headline dichotomy overstates what coordinates achieve. The paper is otherwise transparent and its ablations are informative, so this is a qualification rather than a fatal flaw. The reader already issued a CONDITIONAL verdict, and my concern reinforces that conditionality without changing the verdict.","tokens_in":5932,"tokens_out":5621,"duration_ms":62368,"concrete_test":"Train coordinate-augmented CNN-LSTM, MC, and SAN on the SPATIAL-EXPLICIT dataset for 200k iterations using the exact coordinate channels and hyperparameters from the released code, and report final accuracies and learning curves. If all remain below 80%, the claim that coordinate concatenation alone enables high accuracy on spatial relations is not supported for explicit relations and should be qualified to implicit comparative/superlative forms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 1 states that two alternative techniques enable models to achieve high accuracy for VQA instances involving spatial relations, with concatenating relative spatial coordinates being one of them. The evidence for the coordinate branch is strongest on SPATIAL-COMPARATIVE and SPATIAL-SUPERLATIVE, where coordinate-augmented models approach 97%. However, on SPATIAL-EXPLICIT, the most direct test of spatial relations, Section 5 reports that no model exceeds 80% accuracy, and Figure 3 shows that adding coordinates to SAN does not improve it on this dataset, while the reported coordinate benefit for CNN-LSTM is only on SPATIAL-SUPERLATIVE. Thus, coordinate concatenation alone is not shown to enable high accuracy on explicit relational statements; the best coordinate-based model (RelNet with coordinates) reaches only 74%, which the paper itself treats as a substantial gap. The conclusion overgeneralizes the coordinate-map result from implicit comparative/superlative relations to spatial language broadly. The FiLM+convolutions branch is better supported, but the dichotomy presented in the abstract and introduction is too strong.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks which architectural ingredients are necessary for VQA models to handle simple spatial language. Using diagnostic data generated with ShapeWorld, it compares CNN-LSTM, SAN, RelNet, FiLM, and MC on three caption types (SPATIAL-EXPLICIT, SPATIAL-COMPARATIVE, SPATIAL-SUPERLATIVE). The authors then perform targeted modifications: adding or removing coordinate maps, replacing fully-connected layers with convolutions, and changing fusion from concatenation to FiLM-style modulation. They conclude that two techniques suffice for high accuracy on spatial-relation VQA: concatenating image features with relative spatial coordinates, and early fusion via feature-wise linear modulation combined with convolutional layers. They also argue that stacked attention and the RelNet relation module do not independently contribute.","tokens_in":6094,"tokens_out":5388,"duration_ms":52947,"significance":"If the claims are appropriately scoped, this is a useful diagnostic result for VQA architecture design: it isolates which components matter for a specific capability and would suggest that simpler models augmented with coordinate maps or FiLM+convolutions can replace more complex attention/relation machinery for this kind of spatial reasoning. The study is well designed in several respects: it uses controlled diagnostic data, runs each experiment three times with min/max shading, performs targeted ablations, and releases code. The main limitation is that the coordinate-map conclusion is not supported on the explicit spatial-relation dataset, and the comparison has some hyperparameter and training-schedule imbalances that make the attribution of gains to specific components less secure.","major_comments":[{"comment":"The abstract and introduction claim that concatenating relative spatial coordinates is one of two techniques that enable models to achieve high accuracy for VQA instances involving spatial relations. This claim is not supported on SPATIAL-EXPLICIT: §5 reports that no model exceeds 80% accuracy there, and Figure 3 shows that adding coordinates to SAN does not improve performance on this dataset. The coordinate evidence is strong only for SPATIAL-COMPARATIVE and SPATIAL-SUPERLATIVE. Please qualify the claim to implicit comparative/superlative relations, or provide additional evidence that coordinate maps help on explicit relational statements.","section":"§1 and §5"},{"comment":"The CNN-LSTM baseline is trained for 100k iterations while all other models are trained for 200k. Since much of the paper's argument is framed as improvement over the CNN-LSTM baseline, this asymmetry could overstate the improvement if the baseline would continue to learn with more training. Please retrain the baseline for 200k iterations, or provide learning curves demonstrating that the baseline has saturated by 100k.","section":"§4"},{"comment":"The text says generic parts are kept the same 'to enable a fair comparison', but the language module sizes differ: RelNet uses a 128-dimensional LSTM, FiLM a 512-dimensional GRU, and SAN uses 256-dimensional attention layers. These differences confound the attribution of performance gains to the core module under study. Please either match these hyperparameters across models or provide a sensitivity analysis showing that the conclusions are robust to these choices.","section":"§3"}],"minor_comments":[{"comment":"The notation 'six/eight spatial relations' is unclear; please state explicitly which relations are available for each dataset and why 'behind' and 'in front of' are excluded from the implicit variants.","section":"§2"},{"comment":"The sentence 'processed by an LSTM, or GRU in case of FiLM, of size 512, or 128 in case of RelNet' is ambiguous; please restructure to make the size for each model explicit.","section":"§3"},{"comment":"The +/–/* markers in Figure 3 and the ablation variants in Figures 4–6 are explained only in the captions; consider adding a legend or defining the markers directly in the figure for readability.","section":"Figures 2 and 3"},{"comment":"Dataset names such as 'SPATIAL -EXPLICIT' contain inconsistent spacing; please unify to a single style, e.g., 'SPATIAL-EXPLICIT'.","section":"Throughout"},{"comment":"Reference [6] is an arXiv e-print; if a peer-reviewed version of ShapeWorld exists, please cite that instead or in addition.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid diagnostic study with reproducible code and careful ablations. The main revision needed is to scope the central claim so that the coordinate-map result is not presented as applying to explicit relational statements, where the evidence is absent. The training-schedule and hyperparameter imbalances should also be addressed, at least with sensitivity analysis, before the paper can be recommended for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim overshoots the data. Saying that two techniques 'enable high accuracy for VQA instances involving spatial relations' is too strong when on SPATIAL-EXPLICIT, the most direct test of spatial relations, no model gets above 80%, and coordinate concatenation alone does not even help SAN there. The coordinate result really holds on the implicit comparative/superlative datasets. The other branch—early FiLM-style modulation plus convolutions—is much better supported, and the paper correctly identifies that as the more effective method. So the central dichotomy needs qualification, not abandonment.\n\nWhat the paper does well: it is a clean, controlled diagnostic comparison of SAN, RelNet, FiLM, and MC on ShapeWorld spatial data. The within-model modifications are the right way to attribute performance—adding/removing coordinate maps, swapping fusion and convolution layers—and the three-run averages with min/max shading are clearly reported. The finding that stacked attention and the relation module contribute little for this task, while coordinates and FiLM-style early fusion do, is a useful practical result. The ablations that transfer the FiLM trick to MC and the CNN-LSTM baseline are the strongest part of the paper.\n\nSoft spots worth naming: first, the coordinate overgeneralization mentioned above. Second, the CNN-LSTM baseline trains for 100k iterations versus 200k for every other model; that is a real asymmetry and could make the baseline look weaker than it is. Third, hyperparameters are not perfectly matched across models—RelNet uses a 128-dim LSTM, SAN uses 256-dim attention layers—so exact cross-model comparisons are slightly confounded. These are moderate caveats, not fatal ones. The core attribution story survives.\n\nThis paper is for researchers working on VQA or visual reasoning diagnostics who want to know which architectural pieces actually buy spatial language competence. It is not a landmark, but it is honest and reproducible, with code and public data. I would cite it for the specific finding that relation modules and stacked attention do not drive spatial performance here. If I were editing, I would send it to peer review; the authors should soften the coordinate claim and address the training schedule asymmetry, but the work deserves referee time.","headline":"The coordinate-map branch of the central claim is overstated—coordinates help on implicit spatial comparisons but not on explicit relational statements where no model exceeds 80%—while the FiLM-style fusion result is solid and the ablations are genuinely informative.","tokens_in":6617,"tokens_out":2731,"would_cite":true,"duration_ms":29159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simple spatial language in VQA is powered by coordinate maps or FiLM-style early fusion.","keywords":["visual question answering","spatial relations","feature-wise linear modulation","coordinate maps","diagnostic benchmarks","model ablations","CLEVR","synthetic scenes"],"falsifier":"Retrain the CNN-LSTM baseline for 200k iterations and re-run the ablation; if the extended baseline reduces the gap to FiLM without coordinates or early fusion, the attribution of the gains to those two techniques would be weakened. Alternatively, apply the 'FiLM + convolutions' modification to a spatial dataset with non-uniform object scales, where a single fixed $8 \\times 8$ coordinate map is not translation-invariant.","tokens_in":5709,"feed_emoji":"🧭","tokens_out":6948,"duration_ms":60888,"temperature":0.7,"pith_summary":"The paper asks which architectural ingredient lets a visual question answering (VQA) model understand simple spatial relations like 'left of' and 'above'. Testing four recent models on diagnostic data restricted to spatial statements, it finds that two alternative techniques explain nearly all of the performance gap: attaching relative spatial coordinate maps to the image features, or fusing language and vision early via feature-wise linear modulation (FiLM) in convolutional layers. Stacked attention layers and relation modules contribute little on their own. The result matters because it isolates the load-bearing mechanism behind apparent differences among CLEVR-era models, and shows a plain CNN-LSTM can be boosted to near-FiLM accuracy with the right modification.","feed_headline":"Coordinates or FiLM fusion drive spatial VQA accuracy","feed_subtitle":"Ablation shows attention and relation modules add little—coordinates or FiLM early fusion do the work.","key_machinery":"The two load-bearing mechanisms are the coordinate map and feature-wise linear modulation (FiLM). The coordinate map is a fixed two-channel tensor giving each position of the $8 \\times 8$ image feature grid its relative $(x,y)$ coordinates, concatenated to the features at the start of the model's core module. FiLM lets language features produce per-channel scale and shift parameters that modulate the visual feature map inside convolutional layers; with four stacked $3 \\times 3$ convolutional layers the whole feature grid is covered and local relative positions can be composed into global spatial relations. The paper's evidence comes from targeted ablations—swapping coordinates on and off, replacing fully-connected mixing with convolutions, and replacing concatenation with FiLM fusion—which jointly show that either mechanism alone reproduces the top performance.","core_discovery":"The central claim is that high-level modules—SAN's stacked attention, RelNet's relation module, MC's multimodal core—are not what gives a model spatial language competence. Instead, one of two low-level techniques is responsible: concatenating a fixed map of relative spatial coordinates to the image feature grid, or modulating visual features by language-derived scale and shift parameters (FiLM) inside stacked convolutional layers. The paper shows this by ablating all four models: adding coordinates to MC, SAN, and even the CNN-LSTM baseline lifts them to roughly RelNet's level; removing coordinates from FiLM does not hurt it; and rewiring MC or CNN-LSTM to use FiLM fusion with convolutions (instead of concatenation with fully-connected layers) brings them to approximately FiLM's accuracy. RelNet cannot take the convolution route because its pairwise combinations destroy the two-dimensional arrangement of image features, and SAN's stacked attention does not integrate the changes.","pith_inferences":["If the pattern generalizes beyond synthetic shapes, then comparisons of VQA models on spatial questions should control for whether coordinate maps or FiLM-style fusion are present before crediting attention or relation mechanisms.","A testable extension: on datasets where spatial terms depend on object orientation or reference frames, fixed coordinate maps may be insufficient and the FiLM-plus-convolutions route may need additional geometric priors.","The near-10% gain that coordinates give even to the late-fusion baseline suggests that positional signal can survive average pooling, which could inform lightweight systems that cannot afford full FiLM conditioning.","One should not assume that the two techniques are equivalently robust outside this 64x64 grid setting; rescaling images or using variable-resolution feature maps would be a natural stress test for both mechanisms."],"forward_implications":["On explicit spatial statements, FiLM reaches about 77% accuracy; on comparative and superlative forms it and RelNet reach about 97%, while SAN stays near or slightly above the CNN-LSTM baseline.","Adding relative coordinate maps lifts MC, SAN, and the CNN-LSTM baseline to around the level of RelNet, showing the coordinate trick transfers across architectures.","Replacing the late-fusion CNN-LSTM's core with early FiLM fusion and convolutional layers boosts it to nearly FiLM's accuracy, so the advantage is not tied to the full FiLM architecture.","The relation module of RelNet does not, by itself, improve spatial performance: MC with coordinates matches RelNet with coordinates despite lacking pairwise relation processing.","Four $3 \\times 3$ convolutional FiLM layers span the entire $8 \\times 8$ feature map, so the model can build global spatial judgments from local positional evidence."],"supporting_citations":[{"why":"Supplies the CLEVR diagnostic paradigm and the observation that strong models may not beat baselines, motivating this study.","marker":"[4]"},{"why":"Provides the simulator used to generate the three spatial-language datasets.","marker":"[6]"},{"why":"Introduces the idea of concatenating coordinate maps to image features, which the paper adapts as one of the two key techniques.","marker":"[7]"},{"why":"Contributes the MC model, whose early-fusion and batch-norm design is a comparison point and the target of ablations.","marker":"[8]"},{"why":"Contributes the FiLM modulation layer, the other key technique under study.","marker":"[10]"},{"why":"Contributes the relation module, which the paper ablates and finds non-contributory.","marker":"[11]"},{"why":"Contributes the stacked attention network, also ablated and found non-contributory.","marker":"[13]"}],"fun_headline_variants":["Spatial VQA: coordinates or FiLM, not attention","Coordinates or FiLM fusion drive spatial VQA","For spatial VQA, add coordinates or use FiLM","Spatial VQA: simple low-level cues beat high-level modules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison rests on the CNN-LSTM baseline being trained for only 100k iterations while every other model gets 200k, and on the assumption that keeping hyperparameters uniform across models does not disadvantage any one architecture.","fun_headline_variants_meta":{"raw":{"variants":["Spatial VQA: coordinates or FiLM, not attention","Coordinates or FiLM fusion drive spatial VQA","For spatial VQA, add coordinates or use FiLM","Spatial VQA: simple low-level cues beat high-level modules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1649,"prompt_tokens":822,"completion_tokens":827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":757}},"tokens_in":438,"tokens_out":827,"duration_ms":8913,"temperature":1.0,"reasoning_tokens":757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:48:03.097595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the CNN-LSTM baseline for 200k iterations and re-run the ablation; if the extended baseline reduces the gap to FiLM without coordinates or early fusion, the attribution of the gains to those two techniques would be weakened. Alternatively, apply the 'FiLM + convolutions' modification to a spatial dataset with non-uniform object scales, where a single fixed $8 \\times 8$ coordinate map is not translation-invariant.","supporting_citations":[{"cited_title":"Lawrence Zitnick, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Supplies the CLEVR diagnostic paradigm and the observation that strong models may not beat baselines, motivating this study."},{"cited_title":"An intriguing failing of convolutional neural networks and the CoordConv solution","cited_arxiv_id":null,"evidence_quote":"Introduces the idea of concatenating coordinate maps to image features, which the paper adapts as one of the two key techniques."},{"cited_title":"The visual QA devil in the details: The impact of early fusion and batch norm on CLEVR","cited_arxiv_id":null,"evidence_quote":"Contributes the MC model, whose early-fusion and batch-norm design is a comparison point and the target of ablations."},{"cited_title":"Courville","cited_arxiv_id":null,"evidence_quote":"Contributes the FiLM modulation layer, the other key technique under study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the relation module, which the paper ablates and finds non-contributory."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the stacked attention network, also ablated and found non-contributory."}],"review_version":1}