{"id":"8ff8121e-3f07-4010-8835-d0ad8b50e5e0","arxiv_id":"2507.05677","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"An integrated structural prompt learning method for CLIP reports a state-of-the-art average harmonic mean of 80.70 on 11 base-to-new few-shot classification benchmarks using self- and cross-modal prompt-token interactions and difficulty-based loss weighting.","lead":"The paper introduces a prompt-learning method for CLIP that models relationships between learnable prompts and image or text tokens, within and across modalities, and reweights each sample's loss by difficulty. It reports top average accuracy on standard few-shot benchmarks, but the main equations contain a dimension mismatch and no code is provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. (15)-(16) are undefined under the paper's own hyperparameters: Avt is Lv×N and A_vt^T is N×Lv, so the cross-structural graph convolution cannot be executed as written.","rationale":"The reader's weakest assumption is exactly right: the dimension mismatch in Eqs. (15)-(16) is not a stylistic issue. These equations define the cross-structural graph used in the subsequent graph-convolution equations, so a reader cannot build or run the method from the paper as submitted. This is the most load-bearing concern because it attacks the core contribution directly, not merely the strength of the empirical comparisons. I agree with the REJECT verdict, while noting that the path to conditional acceptance is clear: correct the equations, release code and per-seed results, and add the cited baselines that are currently missing from the tables. Other concerns, such as hyperparameters selected on the same benchmarks and small reported margins over MMA, reinforce but do not replace this central issue.","tokens_in":15242,"tokens_out":6355,"duration_ms":78829,"concrete_test":"Write a conformable implementation of Eqs. (13)-(18) with Lv=4, Lt=6, N=77, and M=196 (or the paper's actual ViT-B/16 shapes). Attempting to evaluate Avt - A_vt^T will raise a shape mismatch. Then, under the most natural corrected reading (for example, row-wise pairwise distances Avv_ij = exp(-β||row_i(Avt) - row_j(Avt)||^2), or whatever the released code uses), rerun the base-to-new protocol and check whether the average HM of 80.70 in Table 1 is reproduced. If no corrected equations or code are supplied, the method remains unimplementable from the text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central cross-structural prompt module depends on affinity matrices Avt ∈ R^{Lv×N} and Atv ∈ R^{Lt×M} (Eqs. 13-14). With the stated Lv=4, Lt=6, and CLIP ViT-B/16 token counts N≈77, M≥196, these matrices are not square. Eq. (15) computes exp(-β || Avt - A_vt^T ||_2^2), but Avt - A_vt^T subtracts a 77×4 matrix from a 4×77 matrix; Eq. (16) has the same problem for Atv. No broadcasting or norm is defined for mismatched matrix subtraction, so Eqs. (17)-(18), which use Avv and Att as graph adjacency matrices, have no well-defined input. The paper provides no code or alternate definition that would disambiguate the intended formula. This is load-bearing because the cross-structural module is one of the two named contributions and is claimed to drive part of the HM gain in Table 4 (80.20→80.35 when CS is added to SS).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Integrated Structural Prompt (ISP) for adapting CLIP to few-shot image classification. ISP consists of three components: a self-structural prompt module that uses cross-attention between learnable prompts and selected frozen tokens, a cross-structural prompt module that builds affinity matrices between prompts and cross-modal tokens and then applies graph convolution to refine prompts, and a sample probing module that assigns per-sample loss weights based on the disagreement between prompted predictions and zero-shot CLIP predictions. The method is evaluated on base-to-new generalization, cross-dataset evaluation, and domain generalization using CLIP ViT-B/16, reporting state-of-the-art harmonic means (e.g., 80.70 on 11 base-to-new datasets) and competitive cross-dataset and domain-generalization accuracies. The paper also includes ablations on prompt lengths, the sample-probing scaling factor, and the number of layers where the modules are applied.","tokens_in":15407,"tokens_out":8225,"duration_ms":94840,"significance":"If the method works as described, the paper would make a useful contribution to prompt learning for vision-language models by explicitly modeling prompt-token and cross-modal structural relationships and by proposing a difficulty-aware loss weighting scheme. The benchmark coverage is broad, and the arithmetic in Tables 1-3 appears internally consistent. However, the current manuscript has a load-bearing technical flaw: the core cross-structural equations, Eqs. (15)-(16), are dimensionally invalid under the paper's own hyperparameters, and no code is supplied to disambiguate the intended computation. Because the cross-structural module is one of the two named technical contributions, the empirical claims cannot currently be verified as written. The self-structural module also contains a shape mismatch in Eq. (10). These issues must be resolved before the results can be accepted.","major_comments":[{"comment":"Eq. (13) defines Avt ∈ R^{Lv×N} and Eq. (15) computes exp(−β∥Avt − A_vt^T∥_2^2). Under the implementation choices in §4.1.2 (Lv=4, Lt=6) and standard CLIP ViT-B/16 token counts, Avt is not square: for example, Lv=4 while N is the text-token count (roughly 77 for a default template), so A_vt^T ∈ R^{N×Lv} and the subtraction Avt − A_vt^T is undefined. The same problem affects Atv in Eq. (16), where Lt=6 and M is the visual token count. Since Eqs. (17)-(18) use Avv and Att as graph adjacency matrices, the entire cross-structural module has no well-defined input as written. The paper provides no code or alternative definition that would disambiguate the intended formula, so the ablation claim in Table 4 that cross-structural prompting contributes gains cannot be checked. Please supply corrected equations and, ideally, a runnable implementation.","section":"§3.4, Eqs. (13)-(16)"},{"comment":"Eq. (10) writes fP_l^v' = CrossAttn(LN(P_l^v), LN(x̃), LN(x̃)) + x̃. If CrossAttn is the standard cross-attention operation with P_l^v as queries and x̃ as keys/values, its output has shape Lv×d_v, whereas x̃ ∈ R^{Lt×d_v}; with Lv=4 and Lt=6, the addition is dimensionally invalid. The surrounding text also first says to select Lv visual tokens and then says to select Lt tokens. This needs to be fixed for the self-structural module to be executable as described.","section":"§3.3, Eqs. (9)-(10)"},{"comment":"Eq. (21) defines α = (2|p(y|x')−p(y|x)| / |p(y|x')+p(y|x)|)^γ. For easy samples, where the zero-shot and prompted probabilities on the ground-truth class are both high and close, α is below 1. Applying max(α,1) then sets the weight to 1, so the promised down-weighting of simple samples does not occur; only hard samples with α>1 are affected. If the intended operation is min(α,1), or clipping to a bounded interval, please state it explicitly and re-run the corresponding ablation.","section":"§3.5, Eq. (21)"}],"minor_comments":[{"comment":"The phrase 'preventing the mode from overfitting' should read 'preventing the model from overfitting.'","section":"Abstract and §3.5"},{"comment":"The name 'CoOpOp' appears to be a typo for 'CoCoOp'; please check all occurrences.","section":"§2 and Table 1"},{"comment":"The reference to 'Eq 5' after Eq. (10) should be to Eq. (9), the cross-attention formula.","section":"§3.3, Eq. (10)"},{"comment":"Under the ViT-B/16 backbone used in the experiments, the visual and text feature dimensions are both 768, so the DCT-based dimensionality reduction in Eq. (12) is vacuous; the motivation and the equation should be clarified or adjusted.","section":"§3.4, Eq. (12)"},{"comment":"The table headers SP, SS, and CS are not defined in the table; in §3.3 SP is used for self-structural prompt, but in the ablation it appears to denote sample probing, which is confusing.","section":"Table 4 and §4.3.1"},{"comment":"The paper reports averages over three seeds but gives no error bars or per-seed results; given the small differences in several comparisons (e.g., Table 3 on ImageNet-V2 and ImageNet-Sketch), some measure of variance would strengthen the claims.","section":"§4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for acceptance in its current form because the central cross-structural equations cannot be executed as written. I do not regard this as an immediate rejection: if the authors can provide corrected equations, an explicit resolution of the shape mismatches, and ideally code or a detailed derivation, the contribution could be salvageable. Without a corrected formulation or code, the reported results should be treated as unverified. I also note that the reported gains from the cross-structural module in Table 4 are small (80.20 to 80.35 HM), so the revision should clarify whether this component is actually responsible for the claimed improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know about 2507.05677: the cross-structural prompt module, one of the two main contributions, is specified with invalid matrix algebra. Eq. (15) computes exp(-beta * ||Avt - Avt^T||^2) where Avt is in R^{Lv x N}; with Lv=4 and N around 77 for ViT-B/16, that subtracts a 4x77 matrix from a 77x4 matrix. Eq. (16) has the same problem. This is not a small typo: the graph convolution in Eqs. (17)-(18) uses these as adjacency matrices, so the central method cannot be executed as written. A reader implementing from the paper would have to guess the intended formula, likely Avt * Avt^T or a pairwise distance matrix.\n\nEverything else in the paper is a fairly clean engineering effort. The self-structural prompt uses top-k token selection plus cross-attention, borrowed from SEP. The cross-structural idea is to use cross-modal affinities to build prompt graphs, and the sample probing reweights the loss by the gap between zero-shot and prompted confidence, similar to AMU. The combination is new at the architecture level, and the reported numbers are internally consistent. An average harmonic mean of 80.70 on base-to-new would be about one point over MMA, a modest but useful gain if it holds.\n\nSoft spots beyond the equations: no code, no per-seed variance (they say three seeds but report only means), and hyperparameters are selected on the test benchmarks. Some directly relevant methods they cite, including SEP, AMU, DePT, LAMM, and ProGrad, are missing from the comparison tables. None of that is fatal if the method is fixed and code is released.\n\nBottom line: as submitted, reject, because the main module is undefined. But this is a likely fixable typo, not a fundamentally broken idea. I would send it to review with instructions for the referee to check the equations, and I would encourage the authors to correct the algebra and release code. The paper is useful for people working on prompt tuning for CLIP, and the benchmark story is worth taking seriously once the method is actually specified. Recommendation: peer-review worthy with major revision, not a desk reject.","headline":"The cross-structural prompt module is defined with invalid matrix subtraction (non-square matrices transposed), so the method cannot be run as written; otherwise this is a competent, incremental prompt-tuning paper with a likely fixable flaw.","tokens_in":16000,"tokens_out":2567,"would_cite":false,"duration_ms":30578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Integrated Structural Prompt (ISP) claims that wiring learnable prompts to frozen tokens, within and across modalities, plus a sample-difficulty loss weight, lifts CLIP's average base-to-new harmonic mean to 80.70% on 11 datasets, above…","keywords":["prompt learning","vision-language models","CLIP","few-shot learning","transfer learning","cross-modal interaction","sample difficulty weighting","structure learning"],"falsifier":"Run the released code or reimplement Eqs. (13)-(18) literally with $L_v=4$, $L_t=6$, $N=77$, and $M=50$: the subtraction in Eq. (15) will fail because $A_{vt}$ is $4 \\times 77$ and $A_{vt}^\\top$ is $77 \\times 4$. That failure settles that the method as written cannot be executed; if a modified operation is used instead, the reported 80.70 average harmonic mean must be reproduced from that modified operation to support the paper's claim.","tokens_in":2183,"feed_emoji":"🧩","tokens_out":3029,"duration_ms":110165,"temperature":0.7,"pith_summary":"This paper argues that CLIP's few-shot transfer improves when learnable prompts are treated as part of a structure rather than as isolated vectors. It introduces self-structural prompt modules, in which prompts attend to the most responsive frozen tokens of the same modality, and cross-structural prompt modules, in which prompts from one modality exchange information with tokens of the other through an affinity graph. A third component, sample probing, reweights each sample's loss by how much the prompted model's confidence differs from zero-shot CLIP's confidence, down-weighting easy samples. The paper reports that this combination raises the average harmonic mean of base and new class accuracy to 80.70% across 11 datasets, with 66.75% average accuracy in cross-dataset evaluation, and it attributes the gains to better prompt–token structure and to reduced overfitting on simple samples.","feed_headline":"Structural prompts lift CLIP few-shot mean to 80.7","feed_subtitle":"Modeling prompt–token structure and sample difficulty edges out prior prompt tuning on 11 benchmarks.","key_machinery":"The load-bearing objects are three modules inserted after frozen CLIP encoder layers. The self-structural prompt is cross-attention between $L$ learnable prompt vectors and the top-response frozen tokens of the same modality, with a two-layer MLP after attention. The cross-structural prompt computes cosine affinity matrices $A_{vt} = \\cos(P^v_l, W_l)$ and $A_{tv} = \\cos(P^t_l, X_l)$, defines prompt-to-prompt similarities $A_{vv} = \\exp(-\\beta\\|A_{vt} - A_{vt}^\\top\\|^2)$ and $A_{tt} = \\exp(-\\beta\\|A_{tv} - A_{tv}^\\top\\|^2)$, and runs one graph-convolution layer over the prompt set to refine $P^v_l$ and $P^t_l$. The sample-probing module computes a per-sample loss weight from the relative gap between the zero-shot CLIP probability and the prompted-model probability, and applies it to the cross-entropy term while leaving the regularization term unchanged. Only the prompts, attention projections, MLPs, and graph-convolution weights are trained; the CLIP backbone stays frozen.","core_discovery":"The central claim is that structural interaction between learnable prompts and frozen tokens, within and across modalities, is a usable and beneficial resource for prompt tuning of CLIP. Within each modality, the self-structural prompt selects the tokens with the largest squared channel responses and applies cross-attention so the learnable prompts absorb class-specific token information while retaining domain knowledge. Across modalities, the cross-structural prompt builds affinity matrices between visual prompts and text tokens and between text prompts and visual tokens, uses those matrices to define prompt-to-prompt similarities $A_{vv}$ and $A_{tt}$, and applies a single graph-convolution layer to refine both prompt sets. The sample-probing module then sets a per-sample loss weight $\\alpha = (2|p(y|x') - p(y|x)| / |p(y|x') + p(y|x)|)^\\gamma$, clipped to be at least 1, so samples that zero-shot CLIP already handles confidently receive less training pressure. The paper reports an 80.70 average harmonic mean on the 11-dataset base-to-new benchmark, 66.75% cross-dataset accuracy, and competitive domain generalization, with ablations indicating each module contributes.","pith_inferences":["The cross-structural equations as printed are not executable at the stated sizes: with $L_v=4$ and $L_t=6$, $A_{vt}$ is $4 \\times 77$ and $A_{tv}$ is $6 \\times 50$, so $A_{vt} - A_{vt}^\\top$ and $A_{tv} - A_{tv}^\\top$ are undefined; a reproduction attempt must first resolve whether this is a typo, a padding or slicing step, or a different intended operation.","One testable extension is to apply sample probing alone, without the structural modules, on top of CoOp or MaPLe; if it transfers, the difficulty-based reweighting is an independent mechanism, not a byproduct of prompt structure.","The top-K token selection is a sparsity claim; comparing it against random token selection of the same size would show whether the choice of discriminative tokens, rather than the extra parameters, drives the self-structural gain.","The cross-modal affinity construction could be replaced with a dimensionally correct bilinear similarity between prompts and tokens; if that change preserves the reported numbers, the published form of Eqs. (15)-(16) is not the operative mechanism."],"forward_implications":["The sample-probing weight is a drop-in reweighting scheme: it modifies only the loss, and inference continues to use the prompted model's probability $p(y|x)$.","If the reported results reproduce, prompt tuning does not need adapters or new backbone parameters to benefit from cross-modal structure; the gains come from the prompt and token organization.","The cross-dataset result (66.75% average) implies that prompts tuned with structural modules on ImageNet transfer to unseen datasets better than text-only prompt methods such as CoOp and CoCoOp.","The ablation table places the largest single-component gain on the self-structural prompt, which alone raises harmonic mean from 78.31 to 80.20, suggesting token–prompt attention carries most of the benefit.","The FGVC Aircraft base accuracy jump (from 40.57 with MMA to 48.40 with ISP) suggests the structural modules are particularly helpful on fine-grained tasks with difficult classes."],"supporting_citations":[{"why":"Supplies the frozen CLIP backbone, the zero-shot baseline, and the shared embedding space that all prompt methods adapt.","marker":"[31]"},{"why":"Introduces learnable text prompts that replace handcrafted templates; it is the primary baseline and the definition of the prompt-tuning paradigm.","marker":"[55]"},{"why":"Defines the base-to-new and domain generalization evaluation settings with image-conditional prompts, and serves as a baseline.","marker":"[54]"},{"why":"MaPLe is the main multi-modal prompt baseline that transfers text prompts to the visual branch, against which ISP compares directly.","marker":"[23]"},{"why":"MMA provides the strongest prior multi-modal adapter baseline and the closest competitor on harmonic mean and cross-dataset accuracy.","marker":"[42]"},{"why":"KgCoOp is the source of the feature-regularization loss $L_{reg}$ that freezes CLIP features to regularize prompted features.","marker":"[43]"},{"why":"SEP motivates the choice of selecting only discriminative tokens by response value for the self-structural prompt.","marker":"[45]"},{"why":"COMMA is a multi-modal prompt fusion baseline that also regularizes text features with frozen CLIP features.","marker":"[20]"},{"why":"LAMM supplies the hierarchical regularization idea for prompts, text features, and prediction probabilities used in the loss design.","marker":"[9]"},{"why":"AMU is the prior uncertainty-based logit fusion method that sample probing is positioned against and improves upon.","marker":"[36]"}],"fun_headline_variants":["Structural prompt–token modeling lifts CLIP to 80.7","Prompt–token structure edges prompt tuning on 11 benchmarks","ISP: model prompt–token relations, adapt loss per sample","Integrating structural prompts and sample difficulty boosts CLIP"],"cache_read_input_tokens":18048,"weakest_assumption_plain":"The method's cross-structural module assumes the affinity matrices $A_{vt}$ and $A_{tv}$ are square enough that $A_{vt}$ minus its transpose and $A_{tv}$ minus its transpose are valid subtractions; with 4 visual prompts, 6 text prompts, roughly 77 image tokens, and roughly 50 text tokens, $A_{vt}$ is $4 \\times 77$ and $A_{tv}$ is $6 \\times 50$, so Equations (15)-(16) are undefined as written.","fun_headline_variants_meta":{"raw":{"variants":["Structural prompt–token modeling lifts CLIP to 80.7","Prompt–token structure edges prompt tuning on 11 benchmarks","ISP: model prompt–token relations, adapt loss per sample","Integrating structural prompts and sample difficulty boosts CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1249,"prompt_tokens":970,"completion_tokens":279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":586,"tokens_out":279,"duration_ms":4442,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:21:14.405805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code or reimplement Eqs. (13)-(18) literally with $L_v=4$, $L_t=6$, $N=77$, and $M=50$: the subtraction in Eq. (15) will fail because $A_{vt}$ is $4 \\times 77$ and $A_{vt}^\\top$ is $77 \\times 4$. That failure settles that the method as written cannot be executed; if a modified operation is used instead, the reported 80.70 average harmonic mean must be reproduced from that modified operation to support the paper's claim.","supporting_citations":[{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP backbone, the zero-shot baseline, and the shared embedding space that all prompt methods adapt."},{"cited_title":"International Journal of Computer Vision 130(9), 2337--2348 (2022)","cited_arxiv_id":null,"evidence_quote":"Introduces learnable text prompts that replace handcrafted templates; it is the primary baseline and the definition of the prompt-tuning paradigm."},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Defines the base-to-new and domain generalization evaluation settings with image-conditional prompts, and serves as a baseline."},{"cited_title":"In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )","cited_arxiv_id":null,"evidence_quote":"MaPLe is the main multi-modal prompt baseline that transfers text prompts to the visual branch, against which ISP compares directly."},{"cited_title":"In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)","cited_arxiv_id":null,"evidence_quote":"MMA provides the strongest prior multi-modal adapter baseline and the closest competitor on harmonic mean and cross-dataset accuracy."},{"cited_title":"In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )","cited_arxiv_id":null,"evidence_quote":"KgCoOp is the source of the feature-regularization loss $L_{reg}$ that freezes CLIP features to regularize prompted features."},{"cited_title":"SEP: Self-Enhanced Prompt Tuning for Visual-Language Model","cited_arxiv_id":"2405.15549","evidence_quote":"SEP motivates the choice of selecting only discriminative tokens by response value for the self-structural prompt."},{"cited_title":"In: AAAI Conference on Artificial Intelligence ( AAAI )","cited_arxiv_id":null,"evidence_quote":"COMMA is a multi-modal prompt fusion baseline that also regularizes text features with frozen CLIP features."},{"cited_title":"Proceedings of the AAAI Conference on Artificial Intelligence 38(3), 1815--1823 (2024)","cited_arxiv_id":null,"evidence_quote":"LAMM supplies the hierarchical regularization idea for prompts, text features, and prediction probabilities used in the loss design."},{"cited_title":"In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)","cited_arxiv_id":null,"evidence_quote":"AMU is the prior uncertainty-based logit fusion method that sample probing is positioned against and improves upon."}],"review_version":1}