{"id":"bb85643b-7c95-4c89-888b-e6acc3c02530","arxiv_id":"2509.00905","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Spotlighter improves CLIP prompt tuning by selecting top-k visual tokens via a prototype-guided activation score, reporting higher accuracy and faster inference with supposedly only 21 extra parameters.","lead":"A new module for CLIP prompt tuning selects only the most informative visual tokens before classification, aiming to boost accuracy and speed. The method adds a learned prototype memory bank and a two-level ranking step, and the authors report gains across 11 few-shot benchmarks with very few extra parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Only 21 extra parameters is contradicted by the §3.2 architecture (multi-head attention, FFN, linear layer, memory bank); this falsifies the paper's lightweight/efficiency claim.","rationale":"The reader's weakest_assumption focused on token-selection discriminativeness and unsubstantiated efficiency cost. The most decisive issue, however, is the direct contradiction between the '21 extra parameters' claim and the architecture described in §3.2. This is an internal inconsistency, not a matter of differing from consensus. A simple parameter count settles it. The FPS numbers are also suspect because the full CLIP encoder must still run before token selection, so the backbone cost is unchanged. Combined with the reduced-epoch baseline retraining and missing error bars, the paper's central claim is not credible as stated. The reader's REJECT verdict is therefore appropriate, though my emphasis is on the parameter-count falsification rather than the token-ranking assumption.","tokens_in":882,"tokens_out":2117,"duration_ms":98684,"concrete_test":"Implement Spotlighter exactly as in §3.2 for ViT-B/16 (or use the released code), and run: params = sum(p.numel() for p in spotlighter_module.parameters() if p.requires_grad) over all trainable tensors in Eq.8–11 and the memory bank U. Report the exact sum. If it exceeds 21, the abstract's 'only 21 extra parameters' is false and the headline claim must be revised.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that Spotlighter adds only 21 parameters while improving both accuracy and speed. This is load-bearing: it supports the paper's positioning as a lightweight, scalable baseline. But the architecture in §3.2 cannot be implemented with 21 trainable parameters. Eq.8 defines MultiHead(LN(U), LN(toklev_i), LN(toklev_i)) + U, requiring at least four linear projections (Q, K, V, output) and two LayerNorms. Eq.9 requires an FFN with two dense layers and biases. Eq.11 requires a Linear layer whose input concatenates tok_ori_t,i with a prototype-weighted sum, already at least d_in×d_out parameters. The semantic memory bank U∈R^{k×c} of Eq.3–6 is itself trainable; Appendix C says each category maintains five 512-dim prototypes, i.e., 5×512×c parameters (2.56M for ImageNet's 1000 classes). Even a single 512×512 linear layer has 262k parameters, so '21 extra parameters' is not a count of any subset of these modules. The efficiency claim is also unsupported because token pruning occurs after the full CLIP image encoder has produced FI (Eq.1); the backbone forward pass is unchanged, so the reported FPS gains cannot arise from the described token-selection mechanism. The accuracy comparisons are further weakened by Appendix C's reduced training epochs for baselines (CoOp: 15, CLIPFit/PromptKD: 30) and the absence of error bars, making the 'up to 11.19%' improvement not clearly separable from retraining-protocol differences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Spotlighter, a plug-and-play token-selection module for CLIP few-shot classification. Spotlighter computes per-token activation scores from sample-wise and semantic-wise similarities, selects top-k tokens, refines them using a learned class-prototype semantic memory bank and IRM/TRM modules, and trains with multiple auxiliary losses. The headline claims are that it outperforms CLIP by up to 11.19% in harmonic mean accuracy, gains up to 0.8K FPS, and adds only 21 extra parameters. The paper reports experiments on 11 few-shot benchmarks, base-to-novel generalization, cross-dataset transfer, ablations, and efficiency comparisons.","tokens_in":17353,"tokens_out":4939,"duration_ms":60854,"significance":"If the efficiency and parameter-count claims were correct, Spotlighter could be a useful lightweight baseline for prompt tuning. The central conceptual idea, selecting representative tokens after CLIP encoding, is not inherently circular. However, the paper's own equations and appendix contradict the '21 extra parameters' claim, and the reported FPS values are internally inconsistent and physically implausible. The accuracy comparisons are also weakened by non-standard training protocols and missing error bars. Consequently, the stated contribution, as presented, cannot be supported.","major_comments":[{"comment":"The '21 extra parameters' claim is contradicted by the architecture described in the manuscript. Eq. (8) uses a multi-head attention block (Q/K/V/output projections plus LayerNorms), Eq. (9) an FFN, Eq. (11) a linear layer, and the semantic memory bank U is updated as a trainable quantity in Eq. (6). Appendix C further states that each category maintains five 512-dimensional prototypes, which alone gives 5×512×C parameters (2.56 million for ImageNet's 1000 classes). No derivation of the '21' count is provided. This is load-bearing because the paper's lightweight positioning and Table IV depend on it.","section":"§3.2, Eq. (8)-(11); Appendix C"},{"comment":"The efficiency results are internally inconsistent and physically implausible. Table IV reports PromptKD at 12943.34 FPS and 'PromptKD w/ Spotlighter +1813.52', while Table VI reports 131.25K FPS for the toklev_1+2 configuration. Even ignoring the internal mismatch, 131.25K images per second on ImageNet with ViT-B/16 on a single GPU is not credible. Moreover, token selection occurs after the full image encoder has produced FI (Eq. (1)), so pruning tokens after the backbone cannot reduce backbone FLOPs. The claimed FPS gains are therefore unexplained.","section":"Table IV vs Table VI; §3.2"},{"comment":"The comparison is not controlled. Appendix C states that CoOp is reduced to 15 training epochs and CLIPFit/PromptKD to 30 epochs, but it does not state how many epochs Spotlighter is trained. With these protocol differences, the 'up to 11.19%' improvement over CLIP is not clearly separable from retraining effects. Despite claiming results are averaged over three seeds, the paper reports no variance or error bars in any table, so statistical significance cannot be assessed.","section":"Appendix C"},{"comment":"The paper's own Limitations section concedes that performance degrades when discriminative tokens are sparse or class boundaries are highly entangled, particularly in ultra-fine-grained settings. This is precisely the regime where the method's core selection assumption—that top-k activated tokens are the most informative—could fail. Yet the paper claims consistent improvements across 11 datasets, including several fine-grained ones, without providing any quantitative analysis of ranking quality or performance in the acknowledged failure regime. This weakens the generality claim.","section":"Appendix A; §4.2"}],"minor_comments":[{"comment":"Typo: 'Apeendix C' should be 'Appendix C'. Also, Table II's caption uses 'PrompKD'.","section":"§4.1"},{"comment":"The caption contains 'CoOpOp' and the row labels are confusingly repeated ('CoOp' appears twice). Please fix.","section":"Table III"},{"comment":"The StanfordCars prompt is listed as 'a photo of a [CLASS], a type of flowers.'; it should be 'a type of car.'","section":"Table X"},{"comment":"The symbol 'tokrep_r' is not defined; presumably it should be 'tokrep_v'.","section":"Eq. (16)"},{"comment":"The notation 'toklevi' and 'toklev_i' is used inconsistently; please define the tier indexing explicitly.","section":"§3.2"},{"comment":"Grammar: 'Our main contributions are lies in' should be 'Our main contributions lie in'.","section":"Introduction"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The '21 extra parameters' assertion is falsified by the paper's own architecture, and the FPS measurements are internally inconsistent and physically implausible. These issues concern the central efficiency claim and cannot be fixed by local editing. The accuracy comparisons are also not suitably controlled, so the empirical case is not persuasive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core efficiency claim is not credible. The paper says “only 21 extra parameters” while §3.2 describes multi-head attention, an FFN, a linear layer, and a trainable memory bank. Appendix C says each category keeps five 512-dim prototypes, which alone is over 2.5M parameters for ImageNet. That is a load-bearing contradiction, not a typo. The FPS numbers are also physically implausible: 131K FPS on ImageNet with ViT-B/16 on a single 4090 is not achievable, and the pruning happens after the full CLIP encoder, so the described mechanism cannot speed up the backbone forward pass. The stress-test note is correct on both counts.\n\nWhat the paper does well: the idea of scoring visual tokens from both sample-wise and semantic-wise perspectives, with a prototype memory bank and two-level ranking, is a sensible extension of prompt tuning. The paper runs a broad set of experiments across 11 benchmarks and includes ablations for losses, prototype initialization, and token counts. That is genuinely useful work for the few-shot VL subfield.\n\nThe soft spots are serious and proportionate. The accuracy improvements are weakened by Appendix C's unequal retraining: CoOp gets 15 epochs, CLIPFit and PromptKD get 30, while Spotlighter presumably trains longer. No error bars appear in the tables despite the claim of three runs. The paper's own Limitations section concedes performance degrades in ultra-fine-grained settings, which is exactly where the token-selection assumption is most fragile. None of this is fatal to the idea, but it is fatal to the paper as written.\n\nWho gets value: someone working on prompt tuning might want to read this to extract the token-selection and prototype-memory idea and to see what ablations look like. But no one should cite the efficiency numbers or parameter count.\n\nRecommendation: this deserves a serious referee only if the authors are required to correct the parameter count and provide plausible, reproducible efficiency measurements. As is, I would not accept it, but I would not desk-reject it either—it needs a major revision. Send it to review with the expectation that the efficiency claims be withdrawn or properly substantiated.","headline":"The token-selection idea is reasonable but the paper's central efficiency claim ('21 extra parameters', impossible FPS) is contradicted by its own architecture and protocol, so the paper needs major correction before it can be taken seriously.","tokens_in":17805,"tokens_out":1795,"would_cite":false,"duration_ms":23585,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spotlighter claims that keeping only the most activated visual tokens, refined by learned class prototypes, improves both accuracy and speed in CLIP few-shot classification—with just 21 extra parameters.","keywords":["prompt tuning","few-shot classification","CLIP","token selection","semantic prototypes","representative feature mining","efficient inference","vision-language models"],"falsifier":"Take a fine-grained dataset with heavily entangled classes, train Spotlighter with the paper's protocol, and sweep k from small to full retention: the central claim fails if keeping all tokens beats the top-k selection, or if a control with the same losses but randomly chosen tokens matches Spotlighter's accuracy. A separate end-to-end timing measurement that includes the IRM/TRM modules would settle whether the claimed 0.8K FPS gain is real.","tokens_in":16840,"feed_emoji":"🎯","tokens_out":7198,"duration_ms":85009,"temperature":0.7,"pith_summary":"Spotlighter aims to show that prompt tuning for CLIP can be improved by treating most visual tokens as noise. The paper proposes scoring every image token twice—once by its alignment with the text embedding, once by its match to class prototypes in a semantic memory bank—and keeping only the top-scoring tokens for prediction. A two-level ranking then fuses those tokens with the prototypes through tiny mapping modules, so a compact token set carries the classification signal. On 11 few-shot benchmarks the method is claimed to beat CLIP by up to 11.19% in harmonic mean accuracy and to add up to 0.8K frames per second, using only 21 trainable parameters, while also lifting existing methods such as CoOp, MaPLe, CLIPFit, and PromptKD when plugged in. If correct, this would make token-level selection a near-free accuracy and efficiency lever for vision-language prompt tuning.","feed_headline":"21 extra parameters beat CLIP by 11.19% at higher speed","feed_subtitle":"Spotlighter keeps only the most informative visual tokens, refines them with learned prototypes, and runs faster with 21 new parameters.","key_machinery":"The load-bearing mechanism is the activation score computed per visual token: a sample-wise term from image-text similarity plus a semantic-wise term from matching the token against class prototypes in a Semantic Memory Bank. A fixed top-k by summed score selects the only tokens that survive to prediction. The Semantic Memory Bank, seeded from class-name text embeddings and updated by momentum, supplies semantic context that compensates for discarded tokens. A two-level ranking stratifies selected tokens into two tiers, and small Image and Text Representative Mapping modules fuse each tier with its prototype, so a compact, semantically weighted token set replaces the full feature map.","core_discovery":"At the core, Spotlighter claims that in CLIP-based few-shot image classification, most visual tokens are redundant or noisy, and that selective pruning guided by semantics improves both accuracy and speed. Each visual token receives an activation score combining sample-wise similarity to the text feature and semantic-wise similarity to class prototypes stored in a Semantic Memory Bank. Only the top-k tokens are kept; prototypes are updated by momentum and used to compensate for discarded information. The selected tokens are split into two tiers and fused with their matched prototypes through Image and Text Representative Mapping modules, and only these representative tokens are used for the","pith_inferences":["My inference: this activation-ranking recipe should be portable to other contrastive vision-language models, since token redundancy is a general property of patch-token encoders; a quick check on a second model family would show whether the gains are CLIP-specific.","My inference: because prototypes are seeded from class-name text embeddings, the semantic-wise score inherits the quality of CLIP's text descriptions; enriching that seed with attributes or definitions could raise the ceiling on fine-grained classes.","My inference: the token count k is fixed; the paper's own limitation on entangled classes suggests an adaptive k, chosen per sample or per class from the activation distribution, is the next natural variable to test.","My inference: to show the accuracy gain is really due to selection rather than to the extra losses, an equal-cost control that trains the same losses on randomly chosen tokens would isolate the ranking mechanism; the paper's bottom-k experiments partially cover this, but not with an equal training budget."],"forward_implications":["Spotlighter can be attached to existing prompt-tuning and adapter methods, improving both their seen-class and unseen-class accuracy while keeping inference fast.","With only 21 trainable parameters, the method makes token-level pruning a practical accuracy-efficiency lever for CLIP, not a costly architectural change.","Because prediction uses a compact set of representative tokens, the method reduces the scale of cross-modal feature interactions at test time, which is the source of the claimed FPS gain.","The semantic memory bank can compensate for discarded visual regions by injecting prototype knowledge, which is why aggressive pruning does not simply throw away semantics.","The method's gains persist across base, novel, few-shot, and cross-dataset settings, so the selection mechanism generalizes beyond the training classes."],"supporting_citations":[{"why":"Supplies the CLIP image and text encoders that Spotlighter is built on and the primary baseline it outperforms.","marker":"Radford et al., 2021"},{"why":"Defines CoOp, the standard prompt-tuning baseline into which Spotlighter is plugged and compared.","marker":"Zhou et al., 2022b"},{"why":"MaPLe, a multi-modal prompt learning baseline; also cited as motivation that full-scale feature interactions are costly.","marker":"khattak et al., 2023"},{"why":"CLIPFit, the parameter-efficient adapter baseline that Spotlighter extends and beats.","marker":"Li et al., 2024a"},{"why":"PromptKD, the strongest baseline used as host for Spotlighter in few-shot and ablation experiments.","marker":"Li et al., 2024c"},{"why":"Prototypical networks, the basis for constructing class prototypes in the semantic memory bank.","marker":"Snell et al., 2017"},{"why":"Grad-CAM, the empirical grounding that intermediate-layer activations localize salient visual concepts, motivating token activation scoring.","marker":"Selvaraju et al., 2017"},{"why":"Classic visualization work underpinning the claim that intermediate activations encode semantically meaningful image regions.","marker":"Zeiler and Fergus, 2014"},{"why":"TextRefiner, a recent prompt-refinement baseline compared in base-to-novel experiments.","marker":"Xie et al., 2024"}],"fun_headline_variants":["Spotlighter: 21 extra params, 11.19% better than CLIP","Token pruning with 21 params: CLIP +11.19% acc","Spotlighter mines representative tokens, beats CLIP by 11%","21 params give CLIP +11.19% acc and +0.8K FPS"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method's accuracy gain depends on the activation score reliably ranking the truly discriminative visual tokens at the top, so that discarding the rest removes noise rather than signal; the authors themselves note this fails in ultra-fine-grained or highly entangled classes.","fun_headline_variants_meta":{"raw":{"variants":["Spotlighter: 21 extra params, 11.19% better than CLIP","Token pruning with 21 params: CLIP +11.19% acc","Spotlighter mines representative tokens, beats CLIP by 11%","21 params give CLIP +11.19% acc and +0.8K FPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3496,"prompt_tokens":719,"completion_tokens":2777,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":2700}},"tokens_in":463,"tokens_out":2777,"duration_ms":24022,"temperature":1.0,"reasoning_tokens":2700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:04:38.882988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fine-grained dataset with heavily entangled classes, train Spotlighter with the paper's protocol, and sweep k from small to full retention: the central claim fails if keeping all tokens beats the top-k selection, or if a control with the same losses but randomly chosen tokens matches Spotlighter's accuracy. A separate end-to-end timing measurement that includes the IRM/TRM modules would settle whether the claimed 0.8K FPS gain is real.","supporting_citations":[],"review_version":1}