{"id":"454b064e-a365-4e40-8ff3-057f3e3b5df5","arxiv_id":"2508.11256","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Decoupling CLIP's self-attention into content and context streams, then distilling context from vision foundation models and diffusion models, yields a CLIP backbone that advances open-vocabulary dense perception across six task families.","lead":"The paper proposes DeCLIP, a training scheme that reworks CLIP's final attention layer into separate \"content\" and \"context\" streams, then fine-tunes CLIP so its local image features become better at dense perception. Across detection, segmentation, 3D, video, and pose benchmarks, the resulting backbone beats existing CLIP-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VIS results are confounded: CLIP-VIS baseline uses ConvNeXt-B, DeCLIP uses ViT-B/L, with no same-backbone vanilla baseline, so the video SOTA gains are not attributable to DeCLIP.","rationale":"I read the paper in good faith and consider the method well-ablated for 2D and 6D tasks. The strongest available evidence for the decoupled distillation is the F-ViT/OV-DQUO detection comparison against CLIPSelf with matching backbone and fine-tuning protocol, and the CAT-Seg semantic segmentation comparison against vanilla CLIP. However, the video experiment, one of the four task families in the headline claim, is the weakest link because the baseline architecture changes. Unlike the other task tables, no same-backbone vanilla control is provided. The reader's conditional verdict already notes this; I agree that the claim is conditional. I am not raising the mechanism concern (proxy tokens/VFM teacher) as the primary load-bearing issue because the empirical ablations, especially the comparison with self-distillation, provide direct evidence that the decoupled design matters even if the mechanistic story is incomplete. The VIS confound is a concrete experimental flaw that, if unaddressed, leaves the 'consistently SOTA across video' part of the central claim unsupported. I therefore recommend keeping the conditional verdict and requiring the same-backbone baseline before the video claim is accepted as stated.","tokens_in":28778,"tokens_out":11305,"duration_ms":123120,"concrete_test":"Run CLIP-VIS with frozen vanilla CLIP ViT-B/16 and ViT-L/14 backbones under the exact LVIS training protocol used in Table 3 (same feature layers [3,5,7,11] and [6,10,14,23], same memory-bank setting), and evaluate on LV-VIS val/test, OVIS, YTVIS19/21, BURST. Compare these numbers to the CLIP-VIS+DeCLIP rows. If the vanilla ViT backbones already achieve AP within the reported DeCLIP gains (or reduce the gap to <2 AP), the claimed video SOTA is not due to DeCLIP, and the central claim should be revised to exclude video or the table replaced with a same-backbone comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 3, the CLIP-VIS baseline uses a ConvNeXt-B backbone, while CLIP-VIS+DeCLIP uses ViT-B/16 and ViT-L/14. The paper states 'Only the backbone network of CLIP-VIS is replaced with DeCLIP,' but this changes both the architecture (ConvNeXt->ViT) and the weights (CLIP->DeCLIP). The reported improvements (e.g., LV-VIS val AP 32.2->34.8 with ViT-B, 37.7 with ViT-L; YTVIS21 AP 42.1->50.6/54.8) could therefore stem from the ViT backbone being better matched to CLIP-VIS's mask-pooling/memory-bank design, not from DeCLIP's decoupled distillation. Since video instance segmentation is explicitly listed in the abstract as one of the task families supporting the 'consistently achieving state-of-the-art' claim, this confound directly undermines a load-bearing part of the central claim. The paper provides no same-backbone vanilla CLIP baseline (e.g., CLIP-VIS with frozen ViT-B/16 or ViT-L/14 using original CLIP weights) to control for the architecture change.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeCLIP, an unsupervised fine-tuning framework that modifies CLIP's visual encoder to improve dense open-vocabulary perception. The core idea is to decouple the final self-attention block into \"content\" and \"context\" streams: context features are distilled to match semantic affinity maps from a VFM (DINOv2), with Stable Diffusion self-attention maps used to complete object boundaries; content features are aligned with CLIP [CLS] representations of image crops, with a region-correlation constraint to preserve dense correlations. The fine-tuned encoder is then dropped into existing open-vocabulary systems for 2D detection/segmentation, training-free segmentation, 3D instance segmentation, video instance segmentation, and 6D pose estimation. The paper reports consistent improvements over baselines, with extensive ablations, sanity checks against the EVA-CLIP default backbone, and sensitivity analyses over VFM choice, dataset source, and hyperparameters.","tokens_in":29155,"tokens_out":8707,"duration_ms":102087,"significance":"If the results hold, DeCLIP would provide a broadly applicable, label-free way to improve CLIP's dense representations, with demonstrated gains across 2D, 3D, video, and pose tasks. The paper is strong in its empirical breadth: code is released, ablations isolate each loss component, sanity checks (Tables 10--12) rule out the EVA-CLIP baseline as the sole source of improvement, and the controlled backbone swaps in detection and segmentation support the central claim. The principal weakness is the uncontrolled comparison in the video experiment, where both the backbone architecture and weights change simultaneously. In addition, the causal 'proxy token' narrative in Section 3.2 is supported only by qualitative attention maps, not by an intervention or quantitative diagnostic; this does not invalidate the method, but the framing is stronger than the evidence. The lack of a same-architecture video baseline is the main load-bearing issue for the paper's broad 'state-of-the-art across a broad spectrum of tasks' claim.","major_comments":[{"comment":"The video instance segmentation comparison is confounded. The CLIP-VIS baseline uses ConvNeXt-B, while CLIP-VIS+DeCLIP uses ViT-B/16 and ViT-L/14. The paper states 'Only the backbone network of CLIP-VIS is replaced with DeCLIP,' but this replacement changes both the architecture (ConvNeXt→ViT) and the weights (original CLIP→DeCLIP). Reported gains (e.g., LV-VIS val AP 32.2→34.8/37.7; YTVIS21 AP 42.1→50.6/54.8) may therefore stem from the ViT backbone being better matched to CLIP-VIS's mask-pooling and memory-bank design, not from the decoupled distillation. Because video is explicitly listed in the abstract as one of the task families supporting the central claim, please add same-backbone baselines: CLIP-VIS with frozen original-CLIP ViT-B/16 and ViT-L/14, using the same feature layers and training settings.","section":"§5.2, Table 3"},{"comment":"The 'proxy token' explanation for CLIP's dense-feature deficits is correlational. The paper claims that proxy tokens cause image tokens to lose semantically relevant attention, but no quantitative measure of proxy-token concentration is provided, and no intervention (e.g., masking or suppressing proxy tokens) is performed to show causality. The ablation in Table 8 does demonstrate that decoupled context distillation improves dense tasks, so the method is not invalidated, but the causal framing in the Introduction and Section 3.2 is stronger than the evidence. Please either soften the causal language or add a quantitative diagnostic/intervention to substantiate the mechanism.","section":"§3.2, Fig. 2"},{"comment":"It is unclear whether the decoupled content/context branches are used only during fine-tuning or also at inference. The text says the self-attention module is 'decoupled' and defines content/context features, but downstream experiments 'replace the backbone' with DeCLIP, which suggests the standard CLIP forward pass is used at test time. If the decoupled branches are training-only auxiliaries, this should be stated explicitly, since the gains would then come through gradient updates rather than architectural changes. If the decoupled branches are used at inference, the paper must specify how downstream methods that expect CLIP's standard ViT interface consume the two streams.","section":"§4.1, Eqs. (4)--(6)"}],"minor_comments":[{"comment":"Several equations (Eqs. (1)--(6)) are rendered with missing glyphs and are not interpretable in the provided version. Please ensure all mathematical symbols are typeset correctly.","section":"§3.1, §4.1"},{"comment":"Clarify which VLM baseline is used for the 'DeCLIP (Ours)' numbers: Table 12 reports 64.9 for OpenAI-CLIP and 64.1 for EVA-CLIP, while Table 7 reports 64.1 without specifying. The caption or table notes should state the default VLM consistently.","section":"Tables 7 and 12"},{"comment":"State explicitly whether the vanilla CLIP-VIS baseline uses a frozen backbone, as the DeCLIP version is described as frozen during training. Also specify the exact feature layers used for the ConvNeXt-B baseline.","section":"§5.2, video experiment"},{"comment":"The sensitivity to dataset size/source is reported only for a subset of tasks; state the evaluated tasks and metrics in the figure caption so readers can judge the scope of the robustness claim.","section":"Fig. 14(c)"},{"comment":"No multiple seeds or confidence intervals are reported. Some gains are on the order of 1--3 mIoU/mAP; reporting variance would help assess significance.","section":"§5.4, Tables 8--12"},{"comment":"The term 'proxy token' is defined only informally. Consider adding a quantitative criterion (e.g., a threshold on attention concentration) or at least a precise operational definition.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well organized and the empirical work is extensive. The central issue is the uncontrolled video backbone comparison in Table 3; if the authors can provide same-backbone vanilla CLIP baselines, the paper may be acceptable after revision. I would also encourage the authors to soften the causal 'proxy token' claim unless they add an intervention. No concerns about scientific integrity were identified; the EVA-CLIP sanity checks and the CC3M sensitivity analysis are commendable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution, not a repackaging. The idea of decoupling the final self-attention block into content and context streams, then distilling context from DINOv2 affinity maps (with SD boundary completion) and content from CLIP's own [CLS] tokens (with a region correlation constraint), is new relative to CLIPSelf, SCLIP, ClearCLIP. The paper earns credit for that.\n\nWhat is done well: extensive ablations (Tables 8-12), sanity checks against EVA-CLIP bias, controlled backbone swaps in detection/segmentation/pose/3D, and code release. The gains are consistent and large, e.g., F-ViT +5.7/5.9 AP on OV-COCO, CAT-Seg +3-4 mIoU, pose AR +5.4. That is convincing evidence that the decoupled distillation is doing real work.\n\nSoft spots. The video instance segmentation results in Table 3 are confounded. The CLIP-VIS baseline uses ConvNeXt-B; DeCLIP uses ViT-B/16 and ViT-L/14. The text says 'only the backbone network is replaced,' but replacing ConvNeXt with ViT changes architecture and pretraining distribution at the same time. Without a same-backbone vanilla CLIP ViT baseline (or at least a CLIP-VIS with frozen ViT-B/16 using original CLIP weights), part of the reported video gains may come from the ViT backbone being a better fit for CLIP-VIS's mask-pooling design, not from DeCLIP. Since the abstract explicitly lists video as one of the tasks supporting the SOTA claim, this is a load-bearing caveat, though not fatal to the whole paper.\n\nSecond, the 'proxy token' mechanism is correlational. The visualizations show CLIP deep layers attend to a few tokens, and VFM doesn't. But there's no isolating experiment that manipulates proxy tokens and shows causal mediation. The method works regardless; the mechanism story is plausible, not proven. Fine as motivation, but the paper should soften the causal language.\n\nMinor: the free parameters (lambda, SD timestep, resolutions, fine-tuned layers) are several, but the sensitivity analysis on lambda is reassuring, and the task coverage makes the method credible.\n\nBottom line: this paper deserves a serious referee. It is a strong empirical contribution with real novelty and careful evaluation. The video confound should be fixed in revision — add a same-backbone baseline or explicitly limit the video claim. I'd cite it and would bring it to the reading group.","headline":"DeCLIP is a genuine, well-ablated contribution to open-vocabulary dense perception, but the video results are confounded by a backbone swap; the core 2D/3D/pose evidence is controlled and convincing.","tokens_in":29635,"tokens_out":2832,"would_cite":true,"duration_ms":30026,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that CLIP's dense-perception failures stem from deep-layer attention collapsing onto a few \"proxy tokens,\" and that a decoupled content/context fine-tuning scheme fixes the features without task labels.","keywords":["open-vocabulary perception","CLIP fine-tuning","decoupled self-attention","knowledge distillation","vision foundation models","diffusion model attention","semantic segmentation","representation learning"],"falsifier":"Visualize the final-layer attention maps of a DeCLIP-fine-tuned model using background patch tokens as queries. If image tokens still concentrate on a few global vertical lines—the proxy-token artifact—rather than on semantically matching regions, then the proposed mechanism is not what drives the reported gains. A complementary check: fine-tune with the context distillation loss replaced by random or permuted VFM affinity maps; if dense perception performance is unchanged, the affinity teacher is not the active ingredient.","tokens_in":28710,"feed_emoji":"👁️","tokens_out":8936,"duration_ms":90596,"temperature":0.7,"pith_summary":"The paper claims that CLIP's poor performance on dense perception tasks—object detection, segmentation, pose estimation—has an identifiable cause: in the deep transformer layers, image tokens stop attending to spatially or semantically related regions and concentrate on a few \"proxy tokens,\" producing features that lack local discriminability and spatial consistency. To fix this, DeCLIP decouples CLIP's final self-attention into a context stream and a content stream, and trains each stream against a separate teacher. The context stream imitates affinity maps from a vision foundation model (VFM), with a diffusion model's self-attention maps used to fill in object boundaries; the content stream aligns pooled region features with CLIP's own image-crop [CLS] representations, plus a VFM region-correlation constraint. The paper reports that this decoupled design avoids the trade-off that arises when both kinds of supervision are applied to a single feature space. If the claim holds, CLIP can serve as a drop-in backbone for open-vocabulary perception in 2D, 3D, and video without task-specific labels.","feed_headline":"CLIP sees pixels better when attention splits into content and context","feed_subtitle":"Unsupervised fine-tuning splits attention to teach CLIP pixel-level semantics for 2D, 3D, and video tasks.","key_machinery":"The key mechanism is the decoupled final attention block. Standard CLIP attention computes an output by attending over values using a softmax of query-key products; DeCLIP rewrites the last block so that a context stream $\\mathbf{z}_{\\text{context}}$ is produced from the query/key projections, and a content stream $\\mathbf{z}_{\\text{content}}$ is produced by aggregating values under the context attention. Context distillation uses a KL divergence between CLIP's token-pair similarity matrix and a VFM affinity matrix that has been element-wise multiplied by fused diffusion-model self-attention maps (SD-Guided Semantic Completion). Content distillation uses a softmax-weighted cosine-similarity","core_discovery":"DeCLIP's central discovery is that CLIP's dense-feature deficiency is not a general lack of local information but a specific attention pathology: starting around the 9th layer, both the [CLS] token and ordinary image tokens concentrate their attention on a few \"proxy tokens,\" so that features for pixels within the same object are not correlated. The fix is to change what the last attention block computes. The block is split so that one output, the context feature $\\mathbf{z}_{\\text{context}}$, is generated from queries and keys and trained to match the pairwise cosine-similarity structure of a VFM's dense features; a second output, the content feature $\\mathbf{z}_{\\text{content}}$, is the va","pith_inferences":["Inference: the same \"proxy-token\" attention pathology may occur in other contrastive vision-language transformers, so the content/context decoupling recipe could be applied to models other than CLIP, provided a suitable VFM teacher is chosen.","Inference: the diffusion-completion step suggests a general design—use one teacher for semantic relations and a second teacher for boundary integrity—that could improve other affinity-distillation pipelines beyond this exact setup.","Inference: downstream methods could deliberately consume only one of the two streams (context for grouping and segmentation, content for recognition), which might yield further gains in training-free pipelines and is directly testable.","Inference: the paper's failure-mode account predicts that the vertical-line artifact in deep CLIP attention maps should disappear in DeCLIP's fine-tuned attention; measuring this artifact on held-out images would be a direct check of the mechanism."],"forward_implications":["Replacing only the CLIP backbone in existing open-vocabulary detectors and segmenters raises novel- and rare-category performance; for example, F-ViT on OV-COCO novel AP50 goes from 37.6 to 43.3 with the ViT-B variant and from 44.3 to 50.2 with ViT-L.","Training-free open-vocabulary semantic segmentation—where pixel-text similarity is used directly—improves from 14.1 average mIoU for vanilla CLIP to 44.1 across eight benchmarks, so no task-specific segmentation head is needed.","3D and video instance segmentation improve when DeCLIP replaces CLIP in OpenMask3D/Open3DIS and CLIP-VIS: on ScanNet200, Open3DIS full AP rises from 23.7 to 26.4, and CLIP-VIS with a ViT-L backbone reaches 49.1 AP on YTVIS21.","Because DeCLIP is trained only on COCO images without dense labels, the same fine-tuned weights carry over to tasks whose supervision comes from different datasets, including 3D scenes and video sequences.","The decoupled features also improve zero-shot region classification accuracy on COCO panoptic at all tested resolutions, indicating that the enhanced dense features retain vision-language alignment."],"supporting_citations":[{"why":"Base ViT-based vision-language model whose final attention is decoupled and fine-tuned; all downstream evaluation starts from this model.","marker":"[12]"},{"why":"Self-distillation baseline; supplies the image-crop [CLS]-alignment loss that the content stream is built on, and the F-ViT detection comparison.","marker":"[15]"},{"why":"Default vision foundation model teacher; its token-pair cosine affinities are distilled into the context stream.","marker":"[55]"},{"why":"Diffusion model whose self-attention maps are used for SD-Guided Semantic Completion of VFM affinity maps.","marker":"[21]"},{"why":"Provides the layer/head fusion scheme (matrix-chain multiplication) used to combine SD self-attention maps in Eq. 8.","marker":"[61]"},{"why":"Also used for fusing SD attention maps and improving CLIP spatial representation; supports the SD completion design.","marker":"[63]"},{"why":"Shows CLIP self-attention can be modified for training-free dense inference; motivates the decoupled attention formulation.","marker":"[57]"},{"why":"Open-vocabulary detector baseline; replacing its CLIP backbone with DeCLIP is a core transfer evaluation.","marker":"[17]"},{"why":"Open-vocabulary segmentation method; swapping in DeCLIP as the image encoder is the segmentation evaluation.","marker":"[11]"},{"why":"Open-vocabulary video instance segmentation baseline; replacing its CLIP backbone with DeCLIP tests generalization to video.","marker":"[38]"}],"fun_headline_variants":["Splitting CLIP attention unlocks dense perception tasks","DeCLIP: decouple attention to make CLIP see pixel-level details","Why CLIP fails at dense tasks: attention sticks to proxy tokens","Content and context split: the fix for CLIP's weak local features","DeCLIP splits attention to give CLIP local discriminability"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that CLIP's dense-perception failure is caused by deep-layer attention collapsing onto a few proxy tokens, and that imitating a VFM's affinity maps (completed by diffusion attention) in the context stream is a sufficient teacher without breaking CLIP's vision-language alignment.","fun_headline_variants_meta":{"raw":{"variants":["Splitting CLIP attention unlocks dense perception tasks","DeCLIP: decouple attention to make CLIP see pixel-level details","Why CLIP fails at dense tasks: attention sticks to proxy tokens","Content and context split: the fix for CLIP's weak local features","DeCLIP splits attention to give CLIP local discriminability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1371,"prompt_tokens":801,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":545,"tokens_out":570,"duration_ms":6167,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:02:39.480775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Visualize the final-layer attention maps of a DeCLIP-fine-tuned model using background patch tokens as queries. If image tokens still concentrate on a few global vertical lines—the proxy-token artifact—rather than on semantically matching regions, then the proposed mechanism is not what drives the reported gains. A complementary check: fine-tune with the context distillation loss replaced by random or permuted VFM affinity maps; if dense perception performance is unchanged, the affinity teacher is not the active ingredient.","supporting_citations":[{"cited_title":"Dinov2: Learning robust visual features without supervision,","cited_arxiv_id":null,"evidence_quote":"Default vision foundation model teacher; its token-pair cosine affinities are distilled into the context stream."},{"cited_title":"Diffusion model is secretly a training-free open vocabulary semantic segmenter,","cited_arxiv_id":null,"evidence_quote":"Provides the layer/head fusion scheme (matrix-chain multiplication) used to combine SD self-attention maps in Eq. 8."},{"cited_title":"Cliper: Hierarchically improving spatial representation of clip for open-vocabulary semantic segmentation,","cited_arxiv_id":null,"evidence_quote":"Also used for fusing SD attention maps and improving CLIP spatial representation; supports the SD completion design."},{"cited_title":"Sclip: Rethinking self-attention for dense vision-language inference,","cited_arxiv_id":null,"evidence_quote":"Shows CLIP self-attention can be modified for training-free dense inference; motivates the decoupled attention formulation."},{"cited_title":"Clip-vis: Adapting clip for open-vocabulary video instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Open-vocabulary video instance segmentation baseline; replacing its CLIP backbone with DeCLIP tests generalization to video."}],"review_version":1}