{"id":"abfd1876-5af8-44ef-8221-c786d778bcbe","arxiv_id":"2412.15277","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PLPP regularizes learnable prompts with a top-k, mutual KL self-distillation loss computed through a no-training LM head, yielding small but consistent gains over PromptSRC.","lead":"This paper adds a perplexity-based loss to prompt learning for vision-language models, where pseudo word labels are obtained from prompt embeddings and a frozen text encoder output. The extra loss acts as self-distillation and produces small accuracy gains over the PromptSRC baseline on classification benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over PromptSRC are marginal and lack error bars or significance tests, so the central claim of superior performance is not established.","rationale":"The reader's verdict is CONDITIONAL and cites lack of error bars as one reason, so my concern aligns partly with that rationale. However, the reader's explicitly identified weakest assumption is the calibration of the LM-head distribution and the validity of top-k truncated KL divergence. I find a different concern more load-bearing: the empirical evidence for the central claim of 'superior performance' is statistically inconclusive. The reported gains over PromptSRC are very small, often below 0.5 percentage points, and no variance is reported despite averaging over only three seeds. In prompt learning, such small differences are within typical seed noise, as evidenced by the seed sensitivity reported in prior work. The cross-dataset result even falls below MaPLe, directly contradicting an unqualified 'superior' claim. The theoretical issues with Eq. 4-5 are real but would not by themselves invalidate the method as a practical regularizer; weak statistical evidence, however, would invalidate the paper's main claim. I therefore recommend keeping the CONDITIONAL verdict: the paper should be accepted only if the authors release per-seed results, code, and significance analyses demonstrating that the gains are reproducible and not noise.","tokens_in":12360,"tokens_out":4229,"duration_ms":40761,"concrete_test":"Re-run PromptSRC and PromptSRC+PLPP with the same 10 random seeds on all 11 datasets under the exact protocol (16 shots, ViT-B/16), recording per-seed accuracies. Compute paired per-dataset differences and apply a paired permutation test or Wilcoxon signed-rank test across datasets for each setting (few-shot, base-to-novel, cross-dataset, domain generalization). If the mean paired difference is not significantly greater than 0 (p<0.05) or the effect size is below 0.3 percentage points, the claimed superiority over PromptSRC is not supported. Also release the per-dataset lambda and alpha table and code to reproduce the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PLPP exhibits superior performance compared to existing methods. The evidence is limited to aggregate tables with tiny margins and no variance reporting. In Table 1, the base-to-novel harmonic mean improves from PromptSRC's 79.97 to 80.33 (+0.36). In Table 3, the domain generalization average improves from 60.7 to 60.8 (+0.1). In Figure 2, the reported per-dataset few-shot improvements over PromptSRC range from 0.1% to 1.3%. In Table 2, the cross-dataset average is 66.29, which is below MaPLe's 66.30 and only +0.48 over PromptSRC. The paper states results are averaged over three random seeds, but no standard deviations, confidence intervals, or significance tests are provided. Differences of less than 0.5 percentage points across 11 datasets can easily arise from seed variation in prompt learning. Without per-seed results or a paired comparison, the reader cannot distinguish a real improvement from noise. The authors also acknowledge that PLPP underperforms MaPLe in cross-dataset evaluation, which directly qualifies the 'superior performance' claim. The method's novelty as a perplexity-based regularizer is secondary; if the marginal gains are not statistically reliable, the paper's main contribution is unsupported, regardless of the theoretical interpretation of the loss.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes PLPP, a plug-in regularization term for prompt learning in vision-language models, applied on top of PromptSRC. A perplexity-style loss is computed by (i) matching each learnable prompt vector to the nearest word in the frozen embedding layer via cosine similarity, and (ii) feeding the text through the frozen text encoder and an untrained linear LM head initialized from the transpose of the embedding weight, to obtain a word distribution. The regularizer is added to the standard cross-entropy prompt loss as two exponential KL terms, intended as perplexity and inverted perplexity losses. Experiments are reported on few-shot classification, base-to-novel generalization, cross-dataset transfer, and domain generalization on 11 datasets. The reported average gains over PromptSRC are small (e.g., +0.36 harmonic mean in base-to-novel, +0.1 average in domain generalization), and in cross-dataset evaluation PLPP's average (66.29) is below MaPLe (66.30).","tokens_in":12660,"tokens_out":9498,"duration_ms":78991,"significance":"The contribution is potentially useful if the empirical claims hold: the regularizer adds no trainable parameters, is conceptually simple, and the self-distillation interpretation is appealing. The manuscript is candid about settings and baselines and uses standard benchmarks. However, the main evidence is not statistically grounded, the loss as written is not the perplexity defined in the theory, and the top-k procedure is incompletely specified. As presented, the paper does not establish that PLPP reliably improves over PromptSRC; it provides suggestive but noisy evidence. The significance would be materially strengthened by variance reporting, significance tests, a corrected loss definition, and a complete specification of the top-k and hyperparameter choices.","major_comments":[{"comment":"The loss defined in Eqs. (4)-(5) is not the perplexity of Eq. (2). Equation (3) states that log-perplexity equals KL(Q||P), but Eq. (4) uses the exponential of one-half the KL divergence and symmetrizes the KL terms. The exponential of a KL divergence is not the perplexity, and the 1/2 factor and symmetrization are not motivated. If the final loss is only a heuristic regularizer, the paper should say so; if it is meant to implement perplexity, the definition must be corrected.","section":"Prompt Learning with Perplexity, Eqs. (4)-(5)"},{"comment":"The top-k truncation is underspecified. The text says the largest k values are retained in Q and the saved indexes are used to obtain P, but it does not state whether the truncated Q and P are renormalized to sum to one. KL divergence requires normalized distributions, so the behavior of the regularizer depends on this choice. Please specify the exact operation and, if renormalization is applied, state it explicitly.","section":"Prompt Learning with Perplexity, top-k description"},{"comment":"The empirical support for the central claim is not statistically grounded. All numbers are averages over three seeds, but no standard deviations, confidence intervals, or paired significance tests are reported. The improvements over PromptSRC are small (harmonic mean +0.36 in Table 1, average +0.1 in Table 3, per-dataset few-shot gains of 0.1-1.3 in Figure 2), so they may be within seed noise. To support the claim of superior performance, report per-seed values or error bars and a paired test over the 11 datasets.","section":"Experiments, Tables 1-3 and Figure 2"},{"comment":"The manuscript states that the best λ and α for few-shot and base-to-novel experiments are listed in the appendix, but no appendix is included. Per-dataset hyperparameter selection on test performance would make the comparisons optimistic and uncontrolled. Disclose the selected values and the selection protocol, or use fixed hyperparameters across datasets.","section":"Implementation Details"},{"comment":"The regularizer is only 'perplexity' if the untrained LM head produces a calibrated word distribution P. No calibration check is provided, and because Q and P are both derived from the same frozen text encoder, the self-distillation interpretation is true by construction rather than independent evidence of regularization value. A simple diagnostic, such as comparing the top-k probabilities with empirical token frequencies, would help establish that the loss has the claimed meaning.","section":"Prompt Learning with Perplexity, LM head initialization"}],"minor_comments":[{"comment":"The abstract claims 'superior performance compared to existing methods,' but Table 2 shows PLPP's cross-dataset average (66.29) is below MaPLe's (66.30); the text acknowledges this, but the abstract and conclusion should be qualified accordingly.","section":"Abstract and Conclusion"},{"comment":"The phrase 'consistent performance advantages' is too strong because Table 1 shows PLPP's harmonic mean on FGVCAircraft (39.94) is below PromptSRC's (40.15), and its base accuracy is lower on four datasets.","section":"Base-to-Novel Generalization"},{"comment":"The statement that PLPP 'consistently outperforms all competing methods across all target datasets' is inaccurate for ImageNet-R, where PLPP ties PromptSRC at 77.8, and for the source ImageNet accuracy, where PLPP (71.0) is below PromptSRC (71.3).","section":"Domain Generalization"},{"comment":"There are frequent typographical errors, including 'PropmtSRC' in the Experiments section, 'retrain' for 'retain' in the top-k description, 'Moveover' in the Introduction, and 'data..' in Related Works.","section":"Throughout"},{"comment":"Equations (4) and (5) contain unmatched parentheses, e.g., 'e 1/2 ·KL(Q1||P1))' has an extra closing parenthesis; also 'H(Q, P) = logP P L' in Eq. (3) should be 'H(Q, P) = log PPL'.","section":"Equations (3)-(5)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal, but the evidence is not yet at the standard required for the claimed contribution. The omitted appendix with per-dataset hyperparameters, the lack of variance reporting, and the inconsistency between Eq. (3) and Eqs. (4)-(5) are the most serious issues. I would recommend the editor require the authors to provide the appendix, per-seed results, and a corrected, fully specified loss before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing about this paper: it introduces a genuinely new regularizer for prompt learning. Placing a frozen LM head (initialized from the transpose of the embedding weight) after the text encoder and matching its output against a cosine-similarity-derived label distribution, with top-k truncation and mutual KL, is not something I have seen in CoOp, CoCoOp, MaPLe, PromptSRC, or ProGrad. The self-distillation framing is apt and the plug-in design adds no trainable parameters. Credit where due: the paper is clearly written, the mechanism is concrete, and the gains, while small, are consistent in sign across few-shot and base-to-novel settings.\n\nThe soft spots are real and they sit exactly on the central claim. The paper says PLPP exhibits superior performance, but the evidence is a set of margins that could easily be seed noise: +0.36 harmonic mean over PromptSRC, +0.1 domain generalization average, +0.1 to +1.3 per-dataset few-shot. No error bars, no significance tests, despite averaging over three seeds. The per-dataset lambda and alpha values are tuned and relegated to an omitted appendix, which makes the comparison hard to audit. The loss definition also does not match the stated perplexity: Eq. 2 defines perplexity as e^{-(1/N) sum log P(w_i|w_<i)}, while Eq. 4-5 use e^{1/2 KL(...)} with no explanation of how the exponent and factor relate. Top-k renormalization is mentioned in prose but never specified, and P2 is described as obtained by topk(P) even though P is the distribution being regularized; the round trip is underspecified. The stress-test note is fair: without variance reporting or paired comparisons, the aggregate tables do not establish the headline claim.\n\nThat said, the weaknesses are about evidence quality, not about the mechanism being nonsensical. The self-distillation interpretation is coherent, and the paper itself honestly acknowledges the cross-dataset average falls below MaPLe. I do not think this is a desk reject. It deserves a serious referee and a request for code, per-seed results, and the hyperparameter appendix. If those come back, the central claim becomes testable. I would not build on it myself until the code is out; the gains are too marginal to cite as a reliable improvement.","headline":"PLPP is a clever, low-cost regularizer with a genuinely new penalty, but the reported gains are too small and too poorly quantified to establish superiority over PromptSRC.","tokens_in":13186,"tokens_out":1415,"would_cite":false,"duration_ms":15137,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A perplexity regularizer turns prompt learning into self-distillation and improves vision-language few-shot accuracy.","keywords":["prompt learning","perplexity","self-distillation","vision-language models","CLIP","few-shot classification","domain generalization","regularization"],"falsifier":"Train PLPP on a dataset whose class names are absent from the model's vocabulary, such as invented or non-English tokens, and compare against PromptSRC; if the accuracy gains persist, the regularizer is not relying on vocabulary meaning, and if they vanish, vocabulary coverage is the load-bearing factor. A second check is to measure the KL divergence between Q and P under randomly initialized prompts; if it is already at chance level, the signal is degenerate.","tokens_in":12144,"feed_emoji":"🧩","tokens_out":7020,"duration_ms":53947,"temperature":0.7,"pith_summary":"The paper proposes PLPP, a plug-in regularizer for prompt learning in vision-language models such as CLIP. It adds a perplexity loss that constrains the learnable prompt vectors toward the output distribution of a frozen text encoder, computed by matching prompt vectors to word embeddings for labels and running an untrained LM head for word probabilities. The authors show this perplexity objective is mathematically equivalent to hard-label self-distillation, and they soften the labels and truncate to top-k values to stabilize training and reduce computation. Across few-shot classification, base-to-novel generalization, cross-dataset evaluation, and domain generalization, PLPP improves over existing prompt-learning baselines without adding trainable parameters.","feed_headline":"Perplexity loss lifts prompt learning without new parameters","feed_subtitle":"A plug-in KL term turns prompt tuning into self-distillation, improving few-shot and generalization accuracy.","key_machinery":"The central object is the perplexity loss $L_{PPL}$ built from two distributions: $Q$, obtained by assigning each prompt vector the index of its nearest word embedding via cosine similarity (softened to a top-k distribution), and $P$, the word probability distribution produced by an untrained LM head, a linear layer without bias whose weights are initialized as the transpose of the embedding weights. The loss uses mutual self-distillation, $e^{\\frac{1}{2}KL(Q_1\\|P_1)} + e^{\\frac{1}{2}KL(P_1\\|Q_1)}$, plus an inverted term $L_{IPPL}$ computed from top-k selections in the opposite direction, and is added to the CLIP cross-entropy loss as $\\lambda(\\alpha L_{PPL} + (1-\\alpha)L_{IPPL})$. This machinery ties prompt vectors to the vocabulary geometry of the frozen text encoder and delivers no-extra-parameter regularization.","core_discovery":"PLPP is a plug-in prompt-regularization method that makes the perplexity of a learnable prompt part of the training signal for frozen CLIP-style models. It obtains a label distribution Q by taking cosine similarities between prompt vectors and the embedding layer's word vectors, and a token distribution P by passing the encoded prompt through a linear LM head initialized from the transpose of the embedding weights; the regularizer is a symmetrized, exponentiated KL term between Q and P. The paper's conceptual claim is that this perplexity loss is equivalent to self-distillation, because it aligns the input prompt distribution with the text encoder's output distribution, and that replacing hard labels with top-k soft labels makes training stable and efficient. Experiments on four evaluation settings show consistent gains, with the clearest improvements over the PromptSRC baseline on novel-class generalization and domain-shift datasets.","pith_inferences":["Because the LM head is tied to the embedding matrix, PLPP likely biases prompts toward the manifold of real word embeddings; one testable consequence is that learned prompts become more interpretable as token combinations than prompts trained without it.","The mutual KL can be read as a bidirectional distributional-consistency regularizer; a natural extension is to anneal the top-k size during training to balance stability and signal strength.","The method is defined for a fixed vocabulary, so applying it to multilingual or open-vocabulary CLIP variants would test whether vocabulary coverage of class names is what makes the regularizer effective.","If the self-distillation equivalence is right, the same perplexity objective should help prompt tuning in other frozen-encoder settings, including text-only language models, which the paper does not test."],"forward_implications":["Any existing prompt-learning method can be augmented with this perplexity regularizer without increasing the number of trainable parameters.","The top-k strategy keeps added computation small enough for practical few-shot and larger-scale use.","Improved base-to-novel and domain generalization imply that perplexity regularization reduces prompt overfitting to base-class patterns.","Because the loss is a form of self-distillation, the same idea can be applied at intermediate transformer layers rather than only at the final prompt.","In the reported experiments, PLPP improves over PromptSRC on most datasets for few-shot and novel-class accuracy, making it a drop-in upgrade for that baseline."],"supporting_citations":[{"why":"Supplies the CLIP vision-language backbone whose frozen encoders PLPP regularizes.","marker":"Radford et al. 2021"},{"why":"Introduces CoOp, the prompt-learning setup that PLPP builds on and regularizes.","marker":"Zhou et al. 2022b"},{"why":"Provides PromptSRC, the strongest baseline and the method that PLPP is integrated with for experiments.","marker":"Khattak et al. 2023"},{"why":"Introduces CoCoOp, a baseline for base-to-novel generalization that PLPP is compared against.","marker":"Zhou et al. 2022a"},{"why":"Supplies the perplexity metric and divergence-frontier formulation that the loss is based on.","marker":"Pillutla et al. 2021"},{"why":"Establishes self-distillation, the mechanism the paper claims perplexity loss instantiates.","marker":"Zhang et al. 2019"}],"fun_headline_variants":["Prompt tuning gets a self-distillation boost via perplexity","Perplexity turns prompt learning into self-distillation","New plug-in regularizer: perplexity as self-distillation for VLMs","Perplexity loss rescues prompt tuning from overfitting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The regularizer stands on the premise that the untrained LM head, initialized from the transpose of the embedding weights, outputs a word-probability distribution whose top-k entries meaningfully match the prompt's intended meaning; if that head is not calibrated, the loss is not really perplexity and the reported gains could come from something else.","fun_headline_variants_meta":{"raw":{"variants":["Prompt tuning gets a self-distillation boost via perplexity","Perplexity turns prompt learning into self-distillation","New plug-in regularizer: perplexity as self-distillation for VLMs","Perplexity loss rescues prompt tuning from overfitting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00013,"raw_usage":{"total_tokens":1146,"prompt_tokens":984,"completion_tokens":162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":92}},"tokens_in":600,"tokens_out":162,"duration_ms":2666,"temperature":1.0,"reasoning_tokens":92,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:06:25.168719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PLPP on a dataset whose class names are absent from the model's vocabulary, such as invented or non-English tokens, and compare against PromptSRC; if the accuracy gains persist, the regularizer is not relying on vocabulary meaning, and if they vanish, vocabulary coverage is the load-bearing factor. A second check is to measure the KL divergence between Q and P under randomly initialized prompts; if it is already at chance level, the signal is degenerate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the perplexity metric and divergence-frontier formulation that the loss is based on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes self-distillation, the mechanism the paper claims perplexity loss instantiates."}],"review_version":1}