{"id":"940ef843-b2e2-4111-86a6-1c1c5a7a26fa","arxiv_id":"2608.00726","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A foveated attention readout over frozen patch tokens recovers most of the localized binding signal that global pooling discards, so apparent spatial blindness can be readout-level rather than representational.","lead":"The paper keeps a frozen vision encoder fixed and varies only how its patch tokens are pooled, finding that a lightweight attention-based readout recovers localized color-shape binding information that global image embeddings lose. It matters because it suggests some reported spatial blindness in vision-language models is an interface artifact rather than missing representation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Foveated readout's attention localization is unverified on the synthetic tasks where the headline 93.5% result is obtained; without direct evidence that attention concentrates on the target, the 'localized binding' interpretation is not fully supported.","rationale":"The reader's weakest assumption concerned the oracle mask in Eq. (1), focusing on GQA* box noise and its effect on the absolute oracle upper bound. That concern is real but mainly affects the natural-image experiment, whose relative comparisons are explicitly acknowledged as robust by the reader. The synthetic CSB result, which is the cleanest demonstration of readout-level blindness, does not depend on GQA* boxes; the oracle there is exact. My stress-test found a different, more directly load-bearing gap: the paper never verifies that the foveated readout's attention is actually localized on the target in the synthetic tasks. The term 'foveated' is central to the title and to contribution (2), and the headline 93.5% paired accuracy is attributed to recovering localized binding information. Without an attention-localization check on CSB, a skeptical reader cannot distinguish between true target foveation and a non-specific foreground/background separation that happens to improve accuracy. The oracle comparison shows the information is present in the tokens, so the central claim about readout-level blindness is not overturned; the foveated readout's mechanism is simply unverified. This supports keeping the CONDITIONAL verdict: the paper should be accepted with conditions that the authors release code and data, and additionally report attention-localization metrics on the synthetic tasks. My recommendation is UNCHANGED because the reader's verdict already is CONDITIONAL, and the additional evidence would strengthen but not alter the conclusion.","tokens_in":20878,"tokens_out":18060,"duration_ms":162507,"concrete_test":"Using the trained foveated readout models that produce the reported CSB paired accuracy (e.g., SigLIP, MLP probe), extract the learned attention weights alpha_i for each test image. Compute the target attention mass: the sum of alpha_i over patch tokens whose centers fall within the ground-truth target bounding box, averaged over all test images and seeds. Report this against the chance level (target box area divided by image area, roughly 14px/384px squared ≈ 0.13%). If the mean target attention mass is substantially above chance (e.g., >0.5) and the top-k attended tokens have high IoU with the target mask, the 'foveated' interpretation is directly supported. If the attention mass is low or spread over non-target foreground/background tokens, the mechanism is different and the claims about foveation should be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central synthetic result is that a single-query foveated readout reaches 93.5% paired-counterfactual accuracy on CSB, recovering most of the oracle-accessible signal. The oracle readout (Eq. 1) is defined explicitly by the target region, so its success directly demonstrates that localized information exists in the patch tokens. However, the foveated readout's behavior is never directly checked against the target region on CSB or CSD: the 'Attention Concentration Diagnostics' section reports effective attended patches and maximum attention weight only for the GQA-derived task. If the learned attention on CSB is not actually concentrated on the target object, the 93.5% result could arise from a different mechanism—for example, suppressing the white background or weighting foreground tokens—rather than from selectively recovering the target's color-shape binding. The lure-invariance results (Figure 6) and the low NSR are suggestive but indirect; a fixed query could conceivably learn a broad foreground selection that still outperforms GAP. This matters because contribution (2) is the foveated probing protocol itself, and the abstract's phrase 'recover localized binding information' relies on the readout being genuinely foveated. The oracle result independently shows that localized information is present, so the broad conclusion about readout-level spatial blindness would likely survive, but the mechanism attributed to the foveated readout would need revision if its attention does not target the object.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether apparent spatial blindness in frozen vision foundation models reflects an absence of localized information in the representation or a loss caused by the global readout interface. Keeping the encoder fixed, it compares global average pooling (GAP), the pretrained summary token, a foveated attention readout with a learned or question-conditioned query, and a non-deployable oracle readout that selects patch tokens within an annotated target region. The evaluation covers three tasks: a synthetic color-shape binding task under clutter (CSB), a color-free crowded shape-detection variant (CSD), and a GQA-derived paired localized-color task (GQA*). The main results are that global readouts collapse on paired-counterfactual accuracy under clutter (3.5% for GAP versus 93.5% for the foveated readout and 99.3% for the oracle on CSB), and that on GQA* question-conditioned foveation (17.0%) substantially outperforms global readouts (3.8-4.3%) while the oracle reaches 33.0%. A counterfactual nuisance-to-signal ratio (NSR) is introduced and shown to be high for global readouts and low for selective readouts. The paper concludes that apparent spatial blindness can arise from the global embedding interface rather than from the absence of localized binding information in frozen patch tokens.","tokens_in":21128,"tokens_out":13926,"duration_ms":103339,"significance":"If the central claim holds, the paper makes a valuable conceptual and methodological contribution: it separates readout-level information loss from representational absence, provides a quantitative NSR diagnostic, and demonstrates with an oracle control that patch tokens retain localized binding information that global embeddings discard. The experimental execution is a strength: the encoder is frozen, readout and probe architectures are matched across conditions, results are averaged over five seeds, three encoders are tested, both linear and MLP probes are used, and the counterfactual pairs are matched. The oracle result makes the information-presence claim robust, and the consistent ordering across configurations supports the interface-level interpretation. The main uncertainties concern the mechanistic interpretation of the foveated readout on the synthetic tasks and the consistency of the natural-image oracle procedure; these are fixable but currently load-bearing for parts of the paper's claims.","major_comments":[{"comment":"The oracle readout defined in Eq. (1) is a masked mean over patch tokens of the full image, but the dataset-construction appendix states that 'The oracle input is the target's ground-truth box cropped from the resized image and stretched to 384×384.' These are different operations: Eq. (1) preserves the original image context and token grid, whereas a crop input changes resolution and removes all other objects. The reported 33.0% GQA* oracle paired accuracy is therefore not an upper bound for information accessible from the frozen tokens of the full image, and the statement that 'the frozen spatial representation contains additional object-specific color information' is not directly supported by this comparison. Please reconcile the two descriptions; if the crop procedure was used, report a masked-token oracle on the full image as well, or reinterpret the foveation-to-oracle gap accordingly.","section":"Eq. (1) vs Appendix 'GQA-derived color binding (GQA*)'"},{"comment":"Direct evidence about where the foveated readout looks is reported only for the GQA*-derived task: EffPatches=5.2±1.2, MaxAttn=0.621±0.035, with a learned temperature of 14.2±3.3. The synthetic CSB/CSD foveated readout uses a single learned query with no temperature and no attention diagnostics. Because the headline 93.5% CSB result is obtained in this setting, the claim that the readout is genuinely 'foveated' on the target, rather than suppressing the white background or broadly weighting foreground tokens, is currently unverified. The lure-invariance and NSR results are suggestive but indirect. Please report attention entropy, maximum attention weight, and ideally the overlap between attended patches and the target mask on CSB/CSD; if the attention is not target-concentrated, the 'localized binding' interpretation of the foveated readout should be revised.","section":"'Attention Concentration Diagnostics'; 'Experimental Protocol' / Figure 4"},{"comment":"The comparison between foveated and global readouts is not capacity-matched: the foveated readout adds a learned query q (and, on GQA*, a learned W_q and temperature), while GAP and the summary token are fixed operations on top of the same probe architecture. The large accuracy gap could therefore be attributed, at least in part, to the extra trainable parameters rather than to spatial selection. The oracle result already establishes that localized information exists, but to support the claim that the foveated protocol succeeds because of selective pooling, the authors should include a control readout with comparable added parameters but without spatial selection, for example a learned per-token weighting applied before global averaging or an attention readout whose query is not trained.","section":"'Foveated Readout' and 'Experimental Protocol'"},{"comment":"The counterfactual pairs used for the NSR analysis are generated by swapping the target's color or shape with a matched non-target object, so the matched partner also changes appearance. Eq. (6) assumes that 'token displacements outside the target region are negligible,' which is violated by this partner displacement. The illustrative attenuation factor (M/N)^2 is thus not a complete description of the GAP representation change, since the partner edit contributes an additional displacement outside the target mask. Please either revise the derivation to include both changed token sets or restrict the illustrative claim to the target-token component and note the partner contribution separately.","section":"Eqs. (5)-(7) and Appendix 'Color-shape binding (CSB)'"}],"minor_comments":[{"comment":"The main text states that 'all readouts are near ceiling (≥99.6%) on the clean no-distractor split,' but Appendix Table 1 reports CLIP GAP clean accuracy of 52.8% (linear) and 45.9% (MLP) and CLIP Summary MLP clean accuracy of 96.9% on CSB. This claim should be qualified to the configurations shown in Figure 4 or corrected.","section":"'Main Readout Comparison and Counterfactual Sensitivity' / Table 1"},{"comment":"The question-only baseline is described as 2.3% paired accuracy versus a 1% uniform-chance level, but the GQA* label distribution is imbalanced and paired chance is not estimated under the empirical label prior. Reporting chance under the empirical class distribution would clarify how much of the 2.3% reflects text priors.","section":"'GQA-derived localized color binding (GQA*)'"},{"comment":"Because the synthetic-task query is content-based and shared across images rather than position-based, the term 'foveated' may overstate the degree of spatial localization. Consider renaming the readout 'attention readout' or adding a clarifying sentence that 'foveation' refers to attention pooling over tokens, not to a spatially anchored fovea.","section":"'Foveated Readout'"},{"comment":"The reference for Subramanian et al. should be capitalized as 'ReCLIP' and the related-work sentence 'Reclip: A strong zero-shot baseline for referring expression comprehension' should be harmonized with the citation format.","section":"References / Related Work"},{"comment":"No code or data release is mentioned; for a diagnostic protocol, providing code for the foveated readout and NSR computation would materially improve reproducibility.","section":"Appendix 'Computing Infrastructure'"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal and the core empirical claim is likely to be of broad interest. My major concerns are fixable: reconcile the GQA* oracle procedure, add synthetic-task attention diagnostics or soften the mechanistic claims, and address the capacity-matched baseline. I would not reject on the basis of the current evidence, but the revision should be checked for whether the foveated readout's behavior on CSB/CSD is actually target-concentrated before the 'localized binding' framing is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result is real: for the tasks studied, the bottleneck is the readout, not the frozen encoder. The oracle readout—pooling only tokens inside the annotated target region—reaches 99.3% paired-counterfactual accuracy on the color-shape binding task, while GAP and the summary token sit near chance. That gap is direct evidence that the binding information lives in the patch tokens. The foveated readout recovers most of it (93.5%), which makes the practical point: a lightweight learned selector can access that signal without fine-tuning the encoder.\n\nThe paper does several things well. The oracle upper bound is a clean diagnostic idea. The counterfactual nuisance-to-signal ratio is a useful, interpretable metric, and the signal-dilution derivation in Eqs. (5)-(7) is a nice mechanistic explanation. The evaluation is careful: three encoders, linear and MLP probes, five seeds, matched counterfactuals. The GQA-derived paired task is a sensible natural-image complement, and the authors honestly flag the noise in GQA* boxes and color labels.\n\nThe soft spots are real but not fatal. The stress-test concern is on target: the paper never shows where the foveated readout actually looks on the synthetic tasks. The attention concentration diagnostics (effective patches 5.2, max weight 0.62) are reported only for GQA*. On CSB, a single learned query could in principle succeed by broadly suppressing background or weighting the foreground, without precisely isolating the target. The oracle already proves the information is present, so the central claim survives; but the \"foveated\" reading—that the readout selectively recovers localized binding—is under-supported. The authors should either plot attention maps on CSB/CSD or soften that claim.\n\nTwo more minor gaps: no capacity-matched non-spatial trainable readout as a control, and no released code/data. The former would strengthen the \"it's the spatial selection, not just trainability\" argument; the latter is a reproducibility issue that should be fixed before publication. Neither undermines the main conclusion.\n\nThis paper deserves a serious referee. I'd send it out, with conditions: add synthetic-task attention localization (or revise the mechanism language), release code and data, and ideally add the non-spatial control. The core diagnostic distinction is valuable and will be reused.","headline":"The readout-vs-representation distinction is solid and well-evidenced, but the paper should verify where its foveated readout actually looks on the synthetic tasks before claiming localization.","tokens_in":21719,"tokens_out":3447,"would_cite":true,"duration_ms":28722,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen patch tokens keep color–shape binding that global readouts discard.","keywords":["vision foundation models","readout probing","spatial binding","foveated attention","nuisance-to-signal ratio","frozen patch tokens","GQA localised color binding","color-shape binding"],"falsifier":"If one re-runs the GQA-derived experiment with masks jittered or shifted by a single patch and oracle paired accuracy drops from 33% toward the 4% global level, the apparent token-level information would be an artifact of mask placement. Similarly, if a foveated readout trained without location supervision fails to beat global readouts on a re-annotated GQA-derived set with tight boxes, the claim that question-conditioned selection recovers localized binding would be falsified.","tokens_in":1800,"feed_emoji":"👁","tokens_out":1843,"duration_ms":80391,"temperature":0.7,"pith_summary":"The paper is trying to distinguish two explanations for why frozen vision models fail at spatially localized tasks: the information was never encoded, or it was encoded in patch tokens but thrown away when read out through a single global image vector. It argues that the second explanation can be the right one, and shows that a lightweight foveated readout, which attention-pools final patch tokens around a learned or question-derived query, recovers most of what an oracle readout with access to the ground-truth target region gets. On a synthetic color–shape binding task under clutter, global readouts score 3.2–3.5% paired-counterfactual accuracy, near chance, while the foveated readout reaches 93.5±0.6% against a 99.3% oracle. On a natural-image GQA-derived task, question-conditioned foveation gives about four times the paired accuracy of global readouts. The paper concludes that apparent spatial blindness can be a property of the global embedding interface rather than a representational absence.","feed_headline":"A selective readout hits 93.5% where global pooling gets 3.5%","feed_subtitle":"Frozen patch tokens carry the binding; a question-conditioned query before pooling recovers it.","key_machinery":"The central objects are three readouts applied to frozen patch tokens: global average pooling (uniform mean), the pretrained summary token, and a foveated readout that attention-pools tokens using a single query—a learned vector for vision-only tasks, or a linear map from the frozen question embedding plus a learned temperature for vision-language tasks. The oracle readout, Eq. (1), is the mean over patch tokens overlapping the annotated target region, and it serves as an upper bound on what the frozen tokens carry. The mechanism is the counterfactual nuisance-to-signal ratio (NSR), which measures how much a readout's embedding changes under label-changing target edits relative to label-preserving nuisance edits, together with the dilution identity $\\Delta z_{\\text{gap}} \\approx \\frac{M}{N}\\Delta x_{\\text{tar}}$: global averaging attenuates the localized representation shift by the fraction $M/N$ of affected tokens, while foveated attention preserves it in proportion to the attention mass on the target.","core_discovery":"Holding a frozen vision encoder fixed and changing only the readout, the paper finds that localized binding information—the conjunction of color and shape of one target object among distractors, or the color of a particular same-category instance selected by a question—is present in the final patch tokens. The pretrained summary token and global average pooling both collapse that signal: they are near chance on paired counterfactuals (3.2±0.1% and 3.5±0.2% on CSB) and their embeddings move more under label-preserving nuisance edits than under label-changing target edits, with a counterfactual nuisance-to-signal ratio near 7–15 versus roughly 0.3 for selective readouts. A single-query attention readout trained from the task loss alone, with no location supervision, reaches 93.5±0.6% paired accuracy, and an oracle pooling tokens inside the annotated target region reaches 99.3%, showing that the ceiling is set by the frozen tokens, not the probe. On GQA-derived natural images, question-conditioned foveation reaches 17.0±0.7% paired accuracy versus 3.8–4.3% for global visual readouts and 33.0% for the oracle. The paper's central claim is that readout-level spatial blindness, not representation failure, explains these global-probe failures.","pith_inferences":["A testable extension: train the foveated readout with oracle masks as soft supervision and measure how much of the remaining gap to the oracle closes.","A benchmark-level consequence the authors leave implicit is that compositionality scores obtained through global embeddings may need to be recomputed with selective readouts before they are interpreted as showing representational failure.","The GQA-derived oracle ceiling of 33% is far below the synthetic ceiling; a re-annotation pass with tighter boxes and unambiguous color labels would show how much of that gap is annotation noise rather than token-level absence.","The diagnostic could be carried to mid-layer tokens and to relational predicates such as above/below and left/right to test whether readout-level blindness is a general interface phenomenon rather than specific to attribute binding."],"forward_implications":["Global readouts do not certify representational absence: a near-chance score from a summary token or global average pooling on a localized task should be re-checked with a readout that can select tokens before pooling.","A lightweight readout trained without location supervision can recover most of the oracle-accessible binding signal, so task-specific spatial selection need not require fine-tuning the encoder.","The counterfactual nuisance-to-signal ratio gives a representation-level predictor of readout quality: readouts with NSR near or above 1 can be expected to fail as clutter grows, while NSR below roughly 0.3 tracks near-oracle accuracy.","Question-conditioned foveation improves paired localized color accuracy on natural images by roughly four times over question-independent global vectors, and the same ordering holds across CLIP, SigLIP, and SigLIP 2 with both linear and MLP probes."],"supporting_citations":[{"why":"Supplies the frozen SigLIP encoder on which all main experiments are run.","marker":"Zhai et al. 2023"},{"why":"Provides the frozen CLIP encoder used in the appendix to confirm the same readout ordering.","marker":"Radford et al. 2021"},{"why":"Supplies the SigLIP 2 encoder used to show the readout ordering generalizes beyond the primary model.","marker":"Tschannen et al. 2025"},{"why":"The attention-based set pooling construction that the foveated readout is deliberately based on.","marker":"Lee et al. 2019"},{"why":"Provides GQA scene graphs from which the paired localized color questions are generated.","marker":"Hudson and Manning 2019"},{"why":"Supplies the Visual Genome images used in the GQA-derived task.","marker":"Krishna et al. 2017"},{"why":"Motivates the synthetic color-shape binding task as a conjunction problem requiring focused attention.","marker":"Treisman and Gelade 1980"},{"why":"Frames multi-object binding failures in vision-language models, the phenomenon the paper re-diagnoses as an interface problem.","marker":"Campbell et al. 2024"},{"why":"Provides prior evidence that binding information can survive within one modality even when global cross-modal alignment ignores it.","marker":"Koishigarina, Uselis, and Oh 2025"}],"fun_headline_variants":["Frozen tokens already know binding; global pooling hides it","93.5% via foveated readout vs 3.5% global: readout is the fix","Spatial blindness is a readout artifact, not a frozen-model limit","Attention query recovers localized binding from frozen patch tokens","Global pooling dilutes evidence; one query restores it"],"cache_read_input_tokens":23808,"weakest_assumption_plain":"The oracle's mask in Eq. (1) is assumed to select exactly the patch tokens that determine the label; if the GQA-derived boxes are misaligned or include non-target content, the oracle upper bound and the fovea's gap to it no longer cleanly measure the information in the frozen tokens.","fun_headline_variants_meta":{"raw":{"variants":["Frozen tokens already know binding; global pooling hides it","93.5% via foveated readout vs 3.5% global: readout is the fix","Spatial blindness is a readout artifact, not a frozen-model limit","Attention query recovers localized binding from frozen patch tokens","Global pooling dilutes evidence; one query restores it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1574,"prompt_tokens":1100,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":716,"tokens_out":474,"duration_ms":4837,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:18:04.578315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If one re-runs the GQA-derived experiment with masks jittered or shifted by a single patch and oracle paired accuracy drops from 33% toward the 4% global level, the apparent token-level information would be an artifact of mask placement. Similarly, if a foveated readout trained without location supervision fails to beat global readouts on a re-annotated GQA-derived set with tight boxes, the claim that question-conditioned selection recovers localized binding would be falsified.","supporting_citations":[],"review_version":2}