{"id":"d1c2d62f-c941-42e9-a7fb-e4f00b8524ca","arxiv_id":"2603.22278","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VLMs use dual spatial-binding mechanisms—dominant global layout codes from the vision encoder plus a secondary LM ordering backup—and amplifying the vision codes improves spatial accuracy.","lead":"Vision-language models bind objects to spatial relations mainly via layout signals already present in the vision encoder, spread across background tokens, not just via the language backbone. Amplifying those vision signals fixes a large share of spatial errors without retraining.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Strip interventions may not isolate pure ordering; residual appearance/projector confounds remain the softest point of the dominance claim.","rationale":"The reader correctly isolates the softest load-bearing assumption: that strip patches and probe directions cleanly isolate causal ordering. The dual-mechanism story is otherwise well-supported by last-token patching (Fig. 2), object-vs-strip contrast (Figs. 6–7), and the vision-ablation recovery in intermediate LM layers (Fig. 8). No internal contradiction appears, and the synthetic results are consistent across two models. The remaining risk is precisely the purity of the vision-side causal interventions, especially once the paper moves to composited natural images and claims a practical fix. That concern does not overturn the evidence already presented, so the verdict stays CONDITIONAL rather than moving to REJECT; it also does not warrant upgrading to ACCEPT until the matched-strip control (or equivalent appearance-matched ablation) is shown. Code release and error bars would further tighten confidence but are secondary to this causal-isolation issue.","tokens_in":15173,"tokens_out":579,"duration_ms":6355,"concrete_test":"Construct a matched-strip control: generate clean/counterfactual pairs whose left and right strips are pixel-identical except for the absolute horizontal placement of the same object (or use identical empty-background strips with only object tokens relocated). Re-run the strip-patching IIA of Fig. 7 and the α-amplification of Table 3. If IIA collapses or the % of corrected failures drops substantially relative to the original strip swap, residual appearance/projector confounds are driving the effect and the dominance claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that vision-encoder layout is the dominant causal source of spatial binding rests on strip-aligned interchange interventions (Figs. 5–7, §5.2.2) and linear-probe amplification (§5.4). These are designed so that left/right strips of matching color are swapped, supposedly transferring only ordinal layout. However, the paper never shows that the swapped strip embeddings are appearance-matched beyond color (texture, edge statistics, projector residuals, or absolute-position encodings can still differ). On synthetic data this risk is smaller; on composite What’sUp scenes—formed by merging two control images into three-object layouts the original set never contained (§3.1)—background strips are especially likely to carry non-ordinal cues. If the intervention moves more than pure order, both the “vision dominates / LM is only backup” ranking and the global amplification fix become overstated. The LM-backup ablation (Fig. 9, Table 2) is cleaner but still leaves the primary dominance claim dependent on the strip-patch purity assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that spatial variable binding in VLMs arises from two concurrent mechanisms: (i) a dominant vision-encoder signal that encodes object layout and is distributed globally across visual tokens (including background “strips”), which the LM backbone directly exploits; and (ii) a secondary LM-backbone mechanism that forms content-independent ordering over object-associated visual tokens when vision-derived order is weak or ablated. Evidence comes from last-token residual-stream interchange interventions that separate order vs. attribute layers, linear probes of vision embeddings, strip-vs-object patching, vision-order ablation followed by LM-only patching, and a global amplification of probe directions that corrects a substantial fraction of errors on What’sUp for Qwen2-VL-7B-Instruct and Gemma-3-4b-it.","tokens_in":15452,"tokens_out":1393,"duration_ms":27379,"significance":"If the dual-mechanism account holds, it reframes spatial failures in VLMs as primarily vision-encoder layout problems rather than LM binding failures, and it supplies a concrete, training-free intervention (global amplification of vision ordering directions) that improves naturalistic spatial reasoning. Strengths include a carefully staged causal design—last-token order/attribute dissociation, strip vs. object patching, vision ablation with residual LM backup, and a random-direction control for amplification—replicated across two VLMs and four settings (three synthetic + What’sUp). The work usefully connects LM variable-binding literature to multimodal models and argues for distributed, multi-token interpretability methods. These are genuine contributions for mechanistic VLM research and for practical spatial-reasoning fixes.","major_comments":[{"comment":"§5.2.2 and Figs. 5–7: The dominance claim rests on strip-aligned interchange interventions that swap left/right strips while matching square color, intended to transfer only ordinal layout. The manuscript does not demonstrate that swapped strip embeddings are matched on residual appearance (texture, edges, absolute-position encodings, projector residuals). Object-only patching fails while strip patching succeeds, which is informative, but without appearance-matched strip controls (or embedding-distance / feature-matching diagnostics), the transferred signal may be richer spatial layout rather than content-independent order. This is load-bearing for ranking vision as the primary “ordering” source and for interpreting the §5.4 amplification as pure order enhancement. Please add controls or reframe the claim as vision-derived layout (not necessarily pure ordinal IDs).","section":"§5.2.2, Figs. 5–7"},{"comment":"§3.1 and §5.4: What’sUp experiments use composite three-object scenes formed by merging control pairs that originally contain only two objects. These layouts never appear in the source control set, so background strips and co-occurrence statistics may be unnatural. Because the amplification result and part of the strip-probe evidence are reported on these composites, the paper should either validate that model behavior and probe geometry match unmerged two-object What’sUp (or other natural multi-object data), or clearly limit the naturalistic claim to this constructed distribution. The abstract’s reference to COCO (in the arXiv abstract) is also inconsistent with the body, which uses What’sUp.","section":"§3.1, §5.4"},{"comment":"§5.1–5.3 and all IIA figures: Interchange-intervention curves are reported as averages over 50 pairs with no error bars, confidence intervals, or significance tests. Several conclusions (order layers 20–22; LM backup only in layers ~13–17 after ablation; strip vs. object contrast) depend on where curves cross. Without uncertainty estimates it is hard to judge robustness of layer ranges and of the vision-vs-LM ranking. Please report variability across pairs (and ideally seeds) for the main IIA plots and for Table 3.","section":"§5.1–5.3, Figs. 2, 6–8, Table 3"}],"minor_comments":[{"comment":"Title/terminology drift: arXiv title and abstract use “spatial variable binding,” while the manuscript title and much of the body use “spatial reasoning.” Align terminology throughout.","section":"Title, Abstract"},{"comment":"Table 1 vs. Table 4: Main-text accuracy is aggregated; appendix splits by direction. A brief note in the main text that performance is roughly balanced across L/R/A/B would help.","section":"Table 1, App. A.1"},{"comment":"Eq. (3): The amplification coefficient α ∈ [1,15] is free; state how α was chosen (grid search on a held-out split? fixed default?) and whether results are sensitive to α.","section":"§5.4, Eq. (3)"},{"comment":"Strip width is model-specific (Table 6) but the main text does not define how strip boundaries are chosen for non-grid What’sUp objects beyond bounding boxes. Clarify the operational definition used for strip patching on natural images.","section":"§5.2.2, Table 6, Fig. 33"},{"comment":"Fig. 2 caption and related plots use a fixed color scheme from the Fig. 3 example while averaging over varied colors/shapes; a short reminder in each caption would reduce misreading.","section":"Fig. 2"},{"comment":"Related work on spatial VLM failures and binding (e.g., Campbell et al., Kamath et al.) is cited; a tighter comparison of what is new relative to Assouel et al. (2025) and Kang et al. (2026) on visual symbolic/ordering mechanisms would help position the dual-source claim.","section":"§2, §4.2"}],"recommendation":"major_revision","confidential_remarks":"The causal toolkit is above average for VLM interpretability submissions; the main risk is overclaiming “content-independent ordering” from imperfectly controlled strip swaps and composite What’sUp scenes. If the authors add appearance-matched strip controls (or reframe to layout) and uncertainty estimates, this is a solid contribution. Scope fits a serious CV/ML venue interested in mechanistic analysis. No integrity concerns noted."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: these VLMs do not invent spatial binding mostly inside the LM. The vision encoder already puts layout information into the projected embeddings, and that signal is spread across background strips, not just object tokens. The LM can form its own content-independent ordering on object tokens when the vision signal is ablated, but that is secondary. Last-token patching cleanly separates order layers from attribute layers; strip vs object patching shows why object-only interventions fail; vision ablation then LM-only patching shows the backup. Amplifying the probe directions on What’sUp corrects a large fraction of errors with a random-direction control. That package is new relative to the LM binding papers and the earlier VLM symbolic work they cite.\n\nWhat they do well is the causal design. Counterfactual swaps, IIA against held-out outcomes, two models, synthetic plus a naturalistic control set, and an explicit random baseline for the intervention. The dual-source story is not just correlational. The distributed-strip finding also forces a real methodological point: single-token analyses will miss the signal.\n\nSoft spots, in proportion. The strip interventions are designed so colors match, but the paper never fully rules out residual texture, edge, or projector artifacts riding along with the swap. That risk is smaller on the synthetic grids and larger on the composited What’sUp scenes (merged pairs that the original control set never contained, horizontal only). So the ranking “vision dominates / LM is only backup” and the size of the amplification win rest on an assumption that is plausible but not airtight. α is free; no error bars or code; abstract says COCO while the body uses What’sUp. Those are real but secondary to the main causal evidence, which still holds on the cleaner synthetic settings.\n\nThis is for people who care about VLM internals and spatial failures. It is not a theory paper and not a new architecture; it is a careful intervention study that reorders priorities toward vision encoders and multi-token interventions. I would send it to peer review. A referee can demand cleaner natural-image controls and uncertainty, but the core result is already worth the time.","headline":"Solid causal evidence that vision-encoder layout codes (strip-distributed, not object-local) dominate spatial binding in two VLMs, with LM ordering as backup; the strip-purity assumption is the real soft spot, not a fatal one.","tokens_in":16038,"tokens_out":537,"would_cite":true,"duration_ms":6427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Vision-language models get most of their spatial binding from global layout signals in the vision encoder, not from the language backbone.","keywords":["vision-language models","spatial reasoning","variable binding","ordering representations","vision encoder","interchange intervention","activation patching","What'sUp"],"falsifier":"On the same clean–counterfactual pairs, if strip-aligned swaps that leave object appearance fixed no longer flip the final answer (or if object-only swaps begin to flip it equally often), the dominance of global vision-layout signals over the language-side backup would be falsified.","tokens_in":16071,"feed_emoji":"👁️","tokens_out":602,"duration_ms":6891,"temperature":0.7,"pith_summary":"Vision-language models must bind objects to properties and spatial relations, yet it has been unclear where those associations are formed. This paper shows they use two concurrent mechanisms. Intermediate layers of the language backbone form content-independent ordering over object tokens, much as language models do for entities in text, but that signal is only a backup. The dominant source is the vision encoder: it already encodes object layout, and that signal is spread across visual tokens, including background strips around the objects, not just the object patches themselves. Causal interchange interventions confirm that swapping those strip-aligned embeddings flips the model’s spatial answers, while object-only swaps do not. Amplifying the vision-derived ordering directions across all image tokens then corrects a large fraction of previously wrong answers on naturalistic scenes, without retraining.","feed_headline":"Spatial binding in VLMs lives mostly in the vision encoder","feed_subtitle":"Layout signals spread into background tokens; amplifying them fixes many spatial errors without retraining","key_machinery":"Strip-aligned interchange intervention on visual embeddings: swapping left/right (or above/below) background-strip tokens between clean and counterfactual images transfers causal ordering information and flips the model’s spatial prediction, whereas patching only object-localized tokens does not.","core_discovery":"VLMs compute spatial variable binding with two concurrent sources of ordering information. The primary source is layout already present in the vision encoder’s embeddings, distributed globally across visual tokens (including surrounding background regions) and directly used by the language backbone. The secondary source is content-independent ordering that the language backbone can form over object-associated tokens when the vision signal is degraded or removed. Strengthening the vision-derived signal improves spatial reasoning on natural images.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Spatial binding in VLMs comes mainly from vision encoder layouts","Vision encoders encode global spatial layouts for VLMs","Dual sources: vision primary, language secondary for VLM binding","Global vision tokens hold main spatial signals in VLMs","Enhancing vision layouts improves VLM spatial binding on COCO"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The claim that strip swaps and linear probe directions isolate pure ordering depends on those interventions not also moving residual appearance, texture, or projector artifacts—especially on composite three-object scenes built from merged real-image pairs.","fun_headline_variants_meta":{"raw":{"variants":["Spatial binding in VLMs comes mainly from vision encoder layouts","Vision encoders encode global spatial layouts for VLMs","Dual sources: vision primary, language secondary for VLM binding","Global vision tokens hold main spatial signals in VLMs","Enhancing vision layouts improves VLM spatial binding on COCO"]},"model":"grok-4.5","effort":"low","cost_usd":0.007416,"raw_usage":{"total_tokens":1800,"prompt_tokens":750,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":74160000,"prompt_tokens_details":{"text_tokens":750,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":967,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":750,"tokens_out":83,"duration_ms":8865,"temperature":1.0,"reasoning_tokens":967,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T20:16:57.313231+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same clean–counterfactual pairs, if strip-aligned swaps that leave object appearance fixed no longer flip the final answer (or if object-only swaps begin to flip it equally often), the dominance of global vision-layout signals over the language-side backup would be falsified.","supporting_citations":[],"review_version":1}