{"id":"a66d2cea-bbfe-44d7-b212-cdc10c4d8467","arxiv_id":"2501.00457","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DPL searches the per-layer prompt length for CLIP with differentiable architecture search, and reports higher few-shot accuracy than fixed-length prompt baselines.","lead":"A new method, DPL, automatically chooses how many prompt tokens to add at each layer of a vision-language model instead of using a fixed prompt length for all layers. The authors report gains over existing prompt-learning baselines on 11 datasets, though the comparison is clouded by a simultaneous change in the attention mechanism.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.60% average gain may come from the switch to cross-attention (Eqs. 4–5) rather than from the automatic prompt-length search; no fixed-length deep-prompt baseline with cross-attention is reported.","rationale":"The reader's weakest_assumption identifies exactly the missing control: DPL switches from self-attention to cross-attention, and no fixed-length deep-prompt baseline with the same cross-attention is reported. That is the single most load-bearing concern because the paper's contribution is the automatic prompt configuration, not the attention modification. If cross-attention alone explains the improvement, the comparison to MaPLe and other self-attention baselines is confounded, and the 2.60% average gain cannot be attributed to the search. The existing ablation (Table 2) is insufficient: it uses a shallow prompt (depth 1), so it cannot separate the effect of depth from the effect of automatic per-layer length selection. A fixed deep cross-attention baseline would settle this directly. Since this is precisely the reader's concern, I agree with the conditional verdict; no further adjustment is needed.","tokens_in":18214,"tokens_out":3690,"duration_ms":32197,"concrete_test":"Run the DPL training stage on all 11 datasets with a fixed prompt configuration: context length 4 in every layer of both the text and image branches, using the same cross-attention (Eqs. 4–5), same 16-shot data, and same hyperparameters as DPL. Compare the average test accuracy to DPL's 81.71. If the fixed-configuration cross-attention baseline is within one standard error (or about 1%) of DPL, the automatic search is not the source of the gain; if DPL leads by more than 1%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DPL's automatic per-layer context-length search yields a 2.60% average accuracy gain over fixed-length deep-prompt baselines. But DPL also replaces the standard self-attention block with cross-attention (Eqs. 4–5) in both the search and training stages, while all baselines (CoCoOp, PLOT, ProGrad, MaPLe) keep self-attention. The only ablation (Table 2) compares DPL to a shallow (depth-1) cross-attention prompt, which does not control for depth or fixed-length deep prompts. Therefore the reported gain may be entirely due to the cross-attention mechanism, not to the automatic selection of context lengths. If a fixed deep prompt (e.g., length 4 at every layer) with cross-attention already matches DPL, the central claim collapses; the search would be adding nothing. This is the weakest link because it directly undercuts the paper's stated contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPL, a two-stage differentiable method for automatically selecting per-layer context lengths of deep continuous prompts for vision-language models like CLIP. In the search stage, a supernet-style model with softmax-weighted context-length options is trained via alternating updates of continuous prompts and architecture parameters alpha, producing an alpha matrix; the argmax per layer yields a discrete prompt configuration. In the training stage, that configuration is trained from scratch with cross-entropy loss and an optional KL distillation term. Experiments on 11 datasets with ViT-B/16 CLIP report an average accuracy of 81.71%, exceeding CoCoOp, PLOT, ProGrad, MaPLe, zero-shot CLIP, and linear probe. The paper also analyzes alpha-matrix convergence, few-shot behavior, and computational complexity.","tokens_in":18502,"tokens_out":6468,"duration_ms":58983,"significance":"If the empirical gains are truly due to the automatic configuration search, DPL is a useful contribution that removes manual per-layer prompt-length and depth design and is conceptually orthogonal to other prompt designs, making it a plausible building block for future methods. The manuscript has several strengths: the problem is well motivated, the differentiable relaxation is a natural extension of DARTS, the benchmarks cover a standard suite of 11 datasets, and the alpha-matrix evolution analysis is thoughtful. However, the headline claim rests on a confounded comparison: DPL also changes the internal attention mechanism from self-attention to cross-attention (Eqs. 4-5), and the only ablation does not control for this change or for the value of the search itself. Two targeted controlled experiments would substantially raise confidence in the central claim.","major_comments":[{"comment":"The reported 2.60% average accuracy gain may be attributable to the change from self-attention to cross-attention rather than to the automatic context-length search. All baselines (CoCoOp, PLOT, ProGrad, MaPLe) use standard self-attention blocks, while DPL uses cross-attention in every transformer block during both the search and training stages. No experiment isolates this change. The authors should add a fixed-length deep prompt baseline with the same cross-attention mechanism (e.g., context length 4 at all layers, no alpha search) and report its test accuracy. If such a baseline matches DPL, the automatic search contributes nothing; if it does not, the contribution of the search is supported. This comparison is load-bearing for the paper's central claim.","section":"§3.1, Eqs. (4)-(5); Table 1"},{"comment":"The only ablation compares DPL to a shallow (depth-1, context-length 16) cross-attention prompt. This does not control for prompt depth or for the homogeneity or heterogeneity of the configuration, so it cannot demonstrate that the automatic heterogeneous configuration is superior to a manually designed deep prompt with the same attention mechanism. The authors should additionally report (i) a fixed-length deep prompt with cross-attention (e.g., length 4 at every layer) and (ii) a randomly selected heterogeneous configuration with matched total number of prompt tokens. Without these controls, the claim that the searched configuration is better than manually designed configurations is not empirically grounded.","section":"§4.4, Table 2"}],"minor_comments":[{"comment":"The caption says the row dimension is related to context length and the column to model depth, but Section 4.2 states the opposite. Please correct the inconsistency.","section":"Figure 2 caption"},{"comment":"\"Boosts the average test accuracy by 2.60%\" is ambiguous. Please specify the comparison baseline (apparently ProGrad at 79.14%, not the average of all baselines) and state whether the value is in percentage points.","section":"Abstract; §4.3"},{"comment":"The stated search-space size 2.81×10^15 does not match the actual candidate set. With t=4 options per layer, ℓ=12 layers per branch, and two independent branches, the total number of configurations is (4^12)^2 ≈ 2.8×10^14; if branches are counted separately, the number is 4^12 ≈ 1.7×10^7. Please correct the numerical value.","section":"Section 6"},{"comment":"The optimization is described as bilevel, but Algorithm 1 uses single-loop alternating updates of alpha and E rather than a nested optimization. DARTS-style approximations are common, but the paper should explicitly acknowledge that this is an approximation to the bilevel optimum, not an exact solution.","section":"§3.1, Algorithm 1; Eqs. (8)-(9)"},{"comment":"\"Hypereparameter\" is a typo for \"hyperparameter\" near Eq. (13). Also, the KL-divergence choice is given with no motivation; a sentence explaining why distillation is added would help.","section":"§3.2"},{"comment":"\"Zero-sho CLIP\" is a typo for \"zero-shot CLIP.\"","section":"§4.3"},{"comment":"The captions contain typos: \"dataets\" and \"exibit\" should be \"datasets\" and \"exhibit.\"","section":"Appendix A.5, Figures 4 and 7"},{"comment":"The \"single-dominant\" definition uses \"≫\" without a quantitative threshold; the later threshold T in the appendix is described in words but not given a value. A precise definition would improve reproducibility.","section":"§4.2, Definition 4.1; Appendix A.2"},{"comment":"DPL+KD has nearly identical average accuracy to DPL (81.74 vs 81.71) and is worse on several datasets (e.g., DTD, Pets, UCF, SUN397). Consider removing the KD variant from the headline results or providing an analysis of when it helps.","section":"Table 1"},{"comment":"The notation [x(l), E(l)] is overloaded; the authors should use explicit concatenation symbols, since the same bracket style is used for both concatenation and indexing elsewhere.","section":"§2.1, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of cs.LG and makes a plausible contribution if the confound is resolved. The two major comments are addressable with additional controlled experiments; without them, the 2.60% claim cannot be taken at face value. Please also ask the authors to correct the search-space count and clarify the comparison baseline used for the headline gain."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper introduces a genuinely new idea—differentiable search over per-layer prompt length for deep prompt learning on CLIP—and backs it with solid, extensive experiments on 11 datasets. But there is a load-bearing confound in the evaluation that needs to be fixed before the headline claim (2.60% average gain over fixed-length baselines) can be trusted.\n\nWhat is actually new: instead of treating context length and prompt depth as manually set hyperparameters, they relax the discrete choice via a softmax over candidate lengths (0/2/4/6 per layer), optimize alpha on validation while prompts on training (bilevel), then read off the argmax configuration and retrain. That is a clean, sensible application of DARTS-style NAS to prompt-length selection, and I don't see it in the cited prompt-learning literature. The cross-attention mechanism (Eqs. 4–5) is a reasonable way to mix different context-length options in the supernet; without it the dimensions wouldn't match.\n\nThe paper also does some things well: it reports standard deviations over 3 runs, includes an ablation (shallow vs. searched configuration), discusses computational cost, and is honest about failure at 1–2 shots. The limitation section is real.\n\nNow the soft spots. The main one is the self-attention/cross-attention switch. DPL uses cross-attention in both the search and training stages; every baseline (CoCoOp, PLOT, ProGrad, MaPLe) uses self-attention. The only ablation (Table 2) compares DPL to a depth-1 cross-attention prompt. That does not rule out the possibility that a fixed-length deep prompt with cross-attention would already give most or all of the 2.60% gain. The central claim—that the search over per-layer lengths is what delivers the improvement—is exactly what the paper does not test. The stress-test note is right, and it is not a minor point; it undercuts the stated contribution.\n\nSecondary issues: the \"optimal configuration is dataset-dependent\" claim is partly tautological since they search per dataset on the validation split. And the KD variant (DPL+KD) does not consistently beat DPL, so the knowledge-distillation part is not clearly motivated. No code is provided, which makes the numbers hard to verify.\n\nWho should read it: anyone working on prompt tuning or PEFT for vision-language models. The method is plausible and the experimental setup is otherwise careful. It deserves a serious referee—not a desk reject—but the referee should ask for a fixed-length prompt with the same cross-attention block, at a couple of depths, plus a random heterogeneous configuration baseline. If those match DPL, the paper becomes a modest contribution about cross-attention prompts, not about automatic design.\n\nMy recommendation: send it to review, but flag the missing ablation as the make-or-break issue.","headline":"The search idea is real and the experiments are careful, but the missing cross-attention ablation means the 2.60% gain cannot yet be attributed to the automatic search.","tokens_in":18930,"tokens_out":2888,"would_cite":false,"duration_ms":25935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic search over per-layer prompt lengths beats fixed prompt designs","keywords":["differentiable prompt learning","continuous prompts","vision-language models","CLIP","prompt configuration search","few-shot learning","bilevel optimization","neural architecture search"],"falsifier":"Train DPL's final-stage model with a fixed context length on every layer (for example, the average searched length) while keeping the cross-attention blocks and all training hyperparameters identical to DPL; if the fixed-length cross-attention model matches DPL's reported 81.71% average accuracy on the 11 datasets, then the automatic search contributed nothing beyond the attention change.","tokens_in":18030,"feed_emoji":"🎯","tokens_out":4601,"duration_ms":42018,"temperature":0.7,"pith_summary":"The paper asks whether the hand-chosen settings of deep prompt learning, one fixed context length applied to every layer, are actually optimal, and it answers no. The authors propose differentiable prompt learning (DPL), a search procedure that automatically picks a context length for the continuous prompts inserted at each transformer layer of CLIP, in the text branch and image branch separately. Using only 16-shot data for the search, DPL finds configurations that vary by dataset and by layer, and these automatic configurations raise average test accuracy by 2.60% on 11 benchmark datasets over strong prompt-learning baselines. This matters because prompt learning is the main parameter-efficient way to adapt frozen foundation models, and its main hyperparameters have so far been set by hand.","feed_headline":"Search finds per-layer prompt lengths that beat fixed designs by 2.6%","feed_subtitle":"DPL automatically tunes CLIP prompt depth per layer, lifting few-shot accuracy on 11 datasets.","key_machinery":"The load-bearing mechanism is a differentiable relaxation of the discrete choice of context length per layer, borrowed from differentiable neural architecture search: each layer has candidate prompt lengths (0, 2, 4, or 6), a trainable alpha weight per candidate is softmaxed into a mixing weight, and the layer output is a convex combination of the outputs under each candidate. To make the candidates mixable despite different lengths, DPL replaces self-attention with cross-attention in the transformer blocks, using the prompt tokens as keys and values while the original tokens provide the queries. The search stage is bilevel, updating alpha on the validation loss and the prompt embeddings on the training loss; after convergence the argmax defines the subprompt used in a standard training stage, optionally with knowledge distillation toward the zero-shot CLIP predictions.","core_discovery":"On the paper's own terms, the discovery is that the optimal continuous-prompt configuration is not uniform: the context length that should be added differs from layer to layer, and the best configuration for the text branch differs from that for the image branch. DPL makes this choice differentiable by relaxing it with a softmax over candidate context lengths, solving a bilevel problem where the prompt vectors minimize training loss and the length-choice parameters minimize validation loss, then committing to the argmax per layer and fine-tuning prompts under that configuration. The paper reports that this automatic design outperforms fixed-length deep-prompt baselines, with the largest gains on datasets with large distribution shift such as EuroSAT and FGVCAircraft, and that the searched configurations are dataset-dependent, supporting the claim that manual homogeneous designs leave performance on the table.","pith_inferences":["The search operates per layer, but nothing restricts it to the four candidate lengths {0, 2, 4, 6}; enlarging the candidate set or allowing per-head prompt lengths could push the same mechanism further, though search cost grows with the space.","The cross-attention modification is itself a design change; the paper does not ablate it against a fixed-length cross-attention baseline, so the relative contribution of the search versus the attention change is an open question a follow-up could settle.","The same differentiable configuration search could be applied to other frozen-foundation adaptation methods, such as visual prompt tuning for detection or prompt tuning for large language models, wherever a discrete hyperparameter controls per-layer inserted vectors.","Dataset-dependent prompt configurations, if they generalize across training seeds, could serve as cheap fingerprints of distribution shift: the per-layer length profile might predict how far a downstream task is from the pretraining distribution."],"forward_implications":["If DPL is right, fixed context length and depth, the two main hyperparameters of deep prompt tuning, need not be chosen by hand; a cheap search over per-layer lengths can replace them.","The dataset-dependence of the found configurations implies that transfer under distribution shift benefits from letting prompt depth adapt to the gap between pretraining and target data, rather than using one global depth.","Because DPL only changes the prompt configuration, it can be combined with other prompt-learning refinements, such as optimal-transport alignment or gradient-aligned distillation, and still contribute its gains.","The asymmetric text and image branch configurations suggest that the two modalities need different amounts of adaptation, a degree of freedom that one-prompt-depth-fits-all methods cannot express.","At inference time the method adds no cost relative to other prompt-tuning methods, since only the chosen prompts remain in the final model."],"supporting_citations":[{"why":"the pre-trained CLIP model that DPL adapts, and the source of the zero-shot predictions used in the knowledge distillation term","marker":"[Radford et al., 2021]"},{"why":"the differentiable architecture-search relaxation that DPL adapts to make context-length choices continuous","marker":"[Liu et al., 2018]"},{"why":"introduces deep continuous prompts into vision transformers, the insertion mechanism DPL generalizes to per-layer lengths","marker":"[Jia et al., 2022]"},{"why":"MaPLe, the deep multi-modal prompt-learning baseline whose fixed-length design DPL outperforms","marker":"[Khattak et al., 2023]"},{"why":"CoOp, the input-only continuous-prompt baseline that establishes the fixed context-length convention DPL challenges","marker":"[Zhou et al., 2022b]"},{"why":"the bilevel-optimization formulation used to structure the search stage","marker":"[Colson et al., 2007]"},{"why":"ProGrad, a baseline whose gradient-aligned knowledge distillation motivates DPL's optional distillation loss","marker":"[Zhu et al., 2023a]"}],"fun_headline_variants":["Automatic per-layer prompt length search beats fixed designs by 2.6%","Differentiable prompt depth learning boosts CLIP accuracy","DPL finds per-layer prompt lengths: 2.6% gain over baselines","Learn prompt lengths per layer to beat uniform designs","CLIP prompt lengths tuned per layer, +2.6% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper credits the accuracy gain to the automatic choice of prompt lengths, but every compared baseline uses self-attention while DPL uses cross-attention in the transformer blocks, and no experiment isolates the attention change from the search, so the search alone is not yet proven to be the source of the improvement.","fun_headline_variants_meta":{"raw":{"variants":["Automatic per-layer prompt length search beats fixed designs by 2.6%","Differentiable prompt depth learning boosts CLIP accuracy","DPL finds per-layer prompt lengths: 2.6% gain over baselines","Learn prompt lengths per layer to beat uniform designs","CLIP prompt lengths tuned per layer, +2.6% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3073,"prompt_tokens":959,"completion_tokens":2114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":575,"tokens_out":2114,"duration_ms":14740,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:51:04.402703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DPL's final-stage model with a fixed context length on every layer (for example, the average searched length) while keeping the cross-attention blocks and all training hyperparameters identical to DPL; if the fixed-length cross-attention model matches DPL's reported 81.71% average accuracy on the 11 datasets, then the automatic search contributed nothing beyond the attention change.","supporting_citations":[{"cited_title":"Maple: Multi-modal prompt learn- ing","cited_arxiv_id":null,"evidence_quote":"MaPLe, the deep multi-modal prompt-learning baseline whose fixed-length design DPL outperforms"},{"cited_title":"An overview of bilevel optimization","cited_arxiv_id":null,"evidence_quote":"the bilevel-optimization formulation used to structure the search stage"}],"review_version":1}