{"id":"9525ac15-4c8e-4e0b-aa1b-b424065255c8","arxiv_id":"2508.20265","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.","lead":"This paper introduces FSA, a training-free plug-in that feeds CLIP's own patch predictions back into its last-layer attention, using output-derived similarities to reweight spatial attention and improve open-vocabulary segmentation. Tested on eight benchmarks with four existing methods and three backbones, it reports small but consistent average gains, with larger gains on weaker baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: FSA may simply be output-space self-smoothing; no baseline applies Af directly to logits, so the attention-feedback claim is unsupported.","rationale":"The reader's weakest-assumption was the linear-additivity of attention isolation in Eq. 6. I agree that this is a real weakness, but I see it as part of a larger, more load-bearing gap: the paper never demonstrates that the improvements are caused specifically by feeding Af into intermediate attention rather than by applying the same learned similarity to the output logits. The isolation concern matters mainly because, if Eq. 6 does not isolate attention, Af is just a class-similarity smoothing operator. The decisive experiment is a logit-space control. This concern does not overturn the empirical results, but it directly targets the central causal claim and the paper's novelty. The existing CONDITIONAL verdict already captures the need for additional validation, so my read leaves the reader's verdict unchanged. I mark agreement as 'partial' because the reader identified a related but narrower issue; I would place the burden on the missing output-space control rather than on the linearity assumption per se.","tokens_in":18590,"tokens_out":11702,"duration_ms":144311,"concrete_test":"Run FSA to compute Af (Eqs. 6–10 and the ensemble in Eq. 14), but instead of modifying Attn_init·V in Eq. 2, apply Af directly to the output predictions: Y_smooth = softmax(Af · softmax(Y_dense)). Also test a logit-space ensemble Y_final = α·Y_dense + (1−α)·Y_smooth with α swept in {0.3,0.5,0.7}. Evaluate on the same 4 baselines × 3 backbones × 8 benchmarks. If Y_smooth alone or the logit-space ensemble recovers most (e.g., ≥80%) of FSA's average mIoU gain over each baseline, then the attention-feedback pathway is not the operative mechanism; the claim should be revised to describe output-space self-smoothing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: re-injecting output-derived patch similarities into the intermediate attention (Eqs. 11–14) is what improves open-vocabulary segmentation. But the evidence for this causal claim is incomplete. FSA computes Af from the output logits (Secs. 4.1–4.2) and then uses Af to modulate V in the attention block. Nothing rules out the simpler hypothesis that FSA is essentially output-space self-smoothing: the same confidence-pruned similarity Af could be applied directly to the output logits (e.g., Y_final = softmax(Af · softmax(Y_dense))) and would likely improve segmentation by propagating confident class predictions among semantically similar patches. No such logit-space baseline is reported. This is not a minor omission: the attention-isolation assumption in Eq. 6 is where the ambiguity enters. Because Eq. 2 contains a residual connection and a nonlinear FFN, Y_dense − Y_dense_uni is not a pure function of Attn_init; it also depends on x and on the interaction between x and the attention output through the FFN. Thus Af may encode little more than 'patches with similar class predictions'—i.e., a self-smoothing kernel. The large gains on weak baselines (MaskCLIP +18.7 mIoU on ViT-L) are exactly what a strong output-space post-processor would produce. Without a logit-space control, the improvements cannot be attributed to attention feedback, which is the paper's stated novelty and the basis of Fig. 1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Feedback-driven Self-adaptive Attention (FSA), a training-free plug-in for CLIP-based open-vocabulary segmentation. FSA computes patch-level semantic similarities from the model's own output logits after an 'attention isolation' step that subtracts a uniform-attention forward pass (Eq. 6), prunes this similarity map using cumulative-confidence thresholding (Eqs. 9-10), and re-injects the resulting feedback attention Af into the last-layer attention block through three adaptations that are ensembled in Eq. 14. The method is evaluated by plugging it into MaskCLIP, SCLIP, ClearCLIP, and ProxyCLIP across ViT-B/L/H, multiple attention types (Q-K, self-self, Proxy with MAE/SAM/DINO), and eight benchmarks. The paper claims consistent improvement across all configurations and datasets.","tokens_in":18882,"tokens_out":5689,"duration_ms":64405,"significance":"If the causal claim holds, FSA is a useful and practical plug-in: it is training-free, has only two fixed hyperparameters, adds modest overhead, and shows consistent average gains, with particularly large improvements on weak baselines such as MaskCLIP. The evaluation is unusually broad — 4 methods, 3 backbones, 8 benchmarks, 3 attention families — and the authors provide code. The main weakness is that the experiments do not include a logit-space control, so the distinctive claim that feedback improves the intermediate attention, rather than simply smoothing output logits, is not yet supported. The paper also overstates consistency, as several individual benchmark cells decrease. These are correctable issues, but they affect the paper's central interpretation and should be addressed before publication.","major_comments":[{"comment":"Missing logit-space control. Af is derived from the output logits and then applied to V or Attninit·V. The reported gains could equally be obtained by treating Af as a spatial smoothing kernel on the output probabilities themselves, e.g., Y_final = softmax(Af · softmax(Y_dense)). Such a baseline would directly test whether the improvement comes from re-injecting feedback into the attention block or from output-space self-smoothing. Without this control, the central claim that FSA 'enhances semantic consistency between internal representations and final predictions' is underdetermined. The large gains on MaskCLIP (+18.7 mIoU on ViT-L) are exactly what a strong output-space post-processor would produce.","section":"Sec. 4.3, Eqs. 11-14"},{"comment":"The attention-isolation assumption is not justified. Because Eq. 2 contains a residual connection and a nonlinear FFN, Y_dense − Y_dense_uni is not a pure function of Attn_init; it also depends on x and on the interaction between the attention output and the FFN. The statement that the subtraction 'allows us to isolate the learned attention's net effect' is therefore an assumption, not a consequence of the equations. The ablation in Table 6 shows that the isolation procedure helps empirically, but it does not verify that the subtracted logits isolate the contribution of the initial attention map. A direct test — for example, perturbing Attn_init and measuring how Y_dense_iso changes, or comparing with a randomized-attention control — would be needed to support the mechanism.","section":"Sec. 4.1, Eq. 6"},{"comment":"The claim of 'consistent improvement across all datasets' is contradicted by the reported numbers. In Table 2, ClearCLIP with ViT-B/16 drops on Context59 (35.9 → 33.8). In Table 3, ProxyCLIP with SAM and ViT-H/14 drops on ADE (22.0 → 21.9). In Table 4, ProxyCLIP K-K with ViT-L/14 drops on Stuff (20.5 → 20.4) and Cityscapes (31.8 → 31.7). Please revise the abstract and Section 5.2 to describe average improvements or to report per-cell exceptions accurately.","section":"Abstract, Sec. 5.2, Tables 2-4"}],"minor_comments":[{"comment":"The text says 'KL divergence, bounded in [0, −∞]'. KL divergence is nonnegative and unbounded above, so the range should be [0, ∞); consequently S = 1/(D+1) lies in (0, 1].","section":"Eq. 8"},{"comment":"Duplicate word: 'minimizes entropy to to adapt models' should be 'to adapt models'.","section":"Sec. 2, Related work"},{"comment":"The main text says FSA adds only 3-5% overhead, but Table S11 reports up to 11.8% overhead for MaskCLIP ViT-B/16 (5.1 → 5.7 ms). Clarify that the 3-5% figure applies to ProxyCLIP, or update the cost claim.","section":"Table 8 vs. Table S11"},{"comment":"Y_dense_uni is described in words but never defined in an equation. Please write the uniform-attention forward pass explicitly, including the softmax normalization, to make the isolation step reproducible.","section":"Sec. 4.1, Eq. 6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is mechanistic: the paper's novelty is attention feedback, but without a logit-space control the evidence does not exclude output-space self-smoothing. The consistency overclaim is easily fixed. I recommend requiring the logit-space baseline and a more careful statement of the isolation assumption before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper has a real and useful idea—feed the model's own patch-level prediction similarities back into the last-layer attention—and it backs it with thorough experiments. The gains on weak baselines are substantial, and the code is released. But the paper overclaims consistency, and the causal mechanism needs a control it doesn't report.\n\nWhat's genuinely new: the feedback loop from output logits to intermediate attention, the attention-isolation trick (subtracting a uniform-attention branch), confidence-based pruning to sparsify the feedback, and the three-way ensemble. None of the cited training-free methods feed output predictions back into attention. The evaluation is broad: four baselines, three backbones, eight benchmarks, three attention families, plus ablations showing each component contributes. Speed overhead is small (3-5%). That's a solid package.\n\nSoft spots, in order of importance.\n\n1. The \"consistently improves\" claim doesn't survive the tables. ClearCLIP on Context59 drops from 35.9 to 33.8 (ViT-B) and from 29.8 to 28.5 (ViT-H). Several other cells show small drops. It's true on average across benchmarks, but not per-dataset. Rewording would fix this.\n\n2. The mechanism is under-supported. The stress-test is right: the paper never applies Af directly to the output logits as a post-processor. If that baseline gave similar gains, the attention-feedback story would be superfluous. Equation 6 assumes Y_dense − Y_dense_uni isolates the attention's contribution, but for methods that keep residual and FFN (MaskCLIP, SCLIP) that subtraction is not a clean linear probe. So we can't rule out output-space self-smoothing. This is the main thing I'd ask for: a logit-space control, perhaps applying Af directly to Y_dense.\n\n3. λ and p are tuned on the eval suite. The sensitivity plot shows they're stable, so it's minor, but still worth acknowledging.\n\nBottom line: this is a useful empirical contribution to the training-free CLIP segmentation subfield. It deserves a serious referee. I'd send it out, but the review should ask for the logit-space control and toned-down consistency claim.","headline":"Genuinely useful plug-in for training-free CLIP segmentation with strong experiments, but the 'consistent improvement' claim is overstated and the mechanism needs a logit-space control to separate attention feedback from output self-smoothing.","tokens_in":19411,"tokens_out":3071,"would_cite":true,"duration_ms":33334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP's own output predictions, fed back into its last-layer attention, improve open-vocabulary segmentation without training.","keywords":["open-vocabulary segmentation","training-free adaptation","CLIP","feedback attention","self-adaptive attention","confidence-based pruning","attention isolation","semantic segmentation"],"falsifier":"Replace the isolated logits Y_iso in Eq. 7 with logits whose patch order has been randomly permuted per image, keeping the marginal class distribution intact; if mIoU gains persist or barely drop, the feedback is not carrying class-coherent spatial information. A second check: on an image set where intermediate attention and final predictions already agree (retention near 100%), FSA should give near-zero improvement; large gains there would indicate it is fixing something other than attention-output inconsistency.","tokens_in":18410,"feed_emoji":"🧩","tokens_out":5744,"duration_ms":57561,"temperature":0.7,"pith_summary":"Open-vocabulary segmentation inherits CLIP's poor patch-level localization: improved intermediate attention often does not reach the final predictions, and attention maps never see class information. This paper proposes a training-free plug-in, FSA, that treats the model's own patch-level class predictions as a spatial coherence prior and feeds pairwise similarities back into the last attention block. The core claim is that this feedback loop makes internal attention and final output semantically consistent, so patches belonging to the same class aggregate more strongly. The paper shows the plug-in improves four existing training-free methods (MaskCLIP, SCLIP, ClearCLIP, ProxyCLIP) on three CLIP backbones and across Q-K, self-self, and proxy attention types on eight benchmarks. If correct, this gives a parameter-preserving way to upgrade any CLIP-based open-vocabulary segmentation pipeline by reusing information the model already computes.","feed_headline":"Training-free feedback from CLIP's own outputs sharpens segmentation","feed_subtitle":"A plug-in module lifts four open-vocabulary segmentation methods across eight benchmarks without retraining.","key_machinery":"The feedback attention Af is the load-bearing object. For every patch pair, it converts the KL divergence between their class-prediction distributions (after uniform-attention isolation) into a similarity, keeps only the highest-confidence entries by cumulative probability threshold p, and scales survivors exponentially with λ before a softmax. It is then inserted into the last attention block as an ensemble, (Af·Attninit + Attninit·Af + Af)/3 · V, so that value aggregation follows semantic cues derived from the model's own outputs rather than only from Q-K or proxy patterns.","core_discovery":"The paper's central claim is that the final patch logits carry the most complete visual and textual semantics, so pairwise KL divergence between those logits is a stronger spatial-coherence prior than any intermediate attention map. To feed that prior back faithfully, the paper isolates the initial attention's contribution by subtracting logits produced by a parallel uniform-attention branch (Eq. 6), then sparsifies the resulting similarity map with a cumulative-confidence threshold and exponential scaling to form Af. Three adaptations—Af applied before, after, or instead of Attninit·V—are ensembled as (Af·Attninit + Attninit·Af + Af)/3 · V. The paper reports this ensemble improves MaskCLIP,","pith_inferences":["A natural extension is to apply the same output-to-attention feedback at multiple layers or iteratively, though the paper reports no gain from iteration; deeper layers may need a different sparsification schedule.","The isolation-by-subtraction trick could be reused in other zero-shot dense tasks such as depth, saliency, or referring segmentation, where CLIP logits are the only supervision, to separate attention's contribution from downstream projections.","The feedback metric need not be KL divergence; replacing it with a learned or calibrated similarity could test whether the gains come from distributional comparison or simply from patch-pair class agreement.","Since FSA only touches the last block, it should compose with fine-tuned or prompt-tuned CLIP variants, giving a test-time adaptation layer that requires no gradient updates."],"forward_implications":["Adding FSA to a training-free CLIP segmentation method requires no training and no change to model weights, so CLIP's zero-shot generalization is preserved.","The reported 3–5% per-image overhead means the feedback loop can serve as a drop-in last-layer upgrade in deployment.","Methods with weak intermediate priors (plain Q-K attention, MaskCLIP) gain the most, suggesting feedback compensates for missing spatial coherence rather than re-ranking already-good attention.","Because the feedback is computed from class predictions, it injects text-aligned class information into the attention layer, which pure self-self or proxy attention cannot do.","Consistent gains across ViT-B/L/H and VFM proxies (MAE, SAM, DINO) indicate the principle transfers across model scales and auxiliary feature sources."],"supporting_citations":[{"why":"ProxyCLIP is the primary training-free baseline and source of the proxy-attention setup and codebase that FSA plugs into; it also motivates modulating the last layer.","marker":"[27]"},{"why":"SCLIP is the self-self attention baseline; it supplies the claim that self-attention disrupts spatial arrangement and the self-self configurations FSA modulates.","marker":"[46]"},{"why":"ClearCLIP is the baseline that identifies residual connections and FFN as noise sources; FSA is validated on its last-layer decomposition.","marker":"[26]"},{"why":"MaskCLIP provides the baseline dense-label extraction from CLIP's value embeddings and is the method FSA improves by the largest margin.","marker":"[67]"},{"why":"CLIP is the frozen vision-text model whose intermediate attention and output logits FSA re-aligns.","marker":"[40]"},{"why":"DINO is the self-supervised ViT used as the default VFM for ProxyCLIP's proxy attention, one of the attention families FSA is tested on.","marker":"[5]"},{"why":"MAE supplies an alternative vision foundation model whose weaker proxy attention FSA improves in the VFM sweep.","marker":"[21]"},{"why":"SAM supplies another auxiliary VFM whose correspondence features form the proxy attention FSA adapts.","marker":"[25]"}],"fun_headline_variants":["Plug-in feedback from CLIP's outputs sharpens segmentation, no training","Training-free self-adaptive attention: CLIP's own logits refine masks","Feedback-driven plug-in boosts open-vocab segmentation across 8 benchmarks","Self-adaptation via output logits: plug-in improves CLIP segmentation","Harnessing final predictions as attention prior enhances CLIP segmentation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that subtracting the uniform-attention branch's logits from the original logits isolates exactly what the learned attention contributes, so the feedback similarity reflects only the initial attention maps and not downstream projection, FFN, or text-alignment effects.","fun_headline_variants_meta":{"raw":{"variants":["Plug-in feedback from CLIP's outputs sharpens segmentation, no training","Training-free self-adaptive attention: CLIP's own logits refine masks","Feedback-driven plug-in boosts open-vocab segmentation across 8 benchmarks","Self-adaptation via output logits: plug-in improves CLIP segmentation","Harnessing final predictions as attention prior enhances CLIP segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1105,"prompt_tokens":763,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":507,"tokens_out":342,"duration_ms":4456,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:12:29.324841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the isolated logits Y_iso in Eq. 7 with logits whose patch order has been randomly permuted per image, keeping the marginal class distribution intact; if mIoU gains persist or barely drop, the feedback is not carrying class-coherent spatial information. A second check: on an image set where intermediate attention and final predictions already agree (retention near 100%), FSA should give near-zero improvement; large gains there would indicate it is fixing something other than attention-output inconsistency.","supporting_citations":[{"cited_title":"Extract free dense labels from clip","cited_arxiv_id":null,"evidence_quote":"MaskCLIP provides the baseline dense-label extraction from CLIP's value embeddings and is the method FSA improves by the largest margin."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP is the frozen vision-text model whose intermediate attention and output logits FSA re-aligns."},{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"DINO is the self-supervised ViT used as the default VFM for ProxyCLIP's proxy attention, one of the attention families FSA is tested on."},{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"MAE supplies an alternative vision foundation model whose weaker proxy attention FSA improves in the VFM sweep."}],"review_version":1}