{"id":"6d73a4cb-8ec2-4166-90eb-df6b3324e4fe","arxiv_id":"2502.07847","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A proposed unified regularizer for CLIP prompt learning, combining Fisher information and confidence penalties, is reported to improve few-shot accuracy and calibration, but the core combined method is not tested and key definitions conflict.","lead":"CalShift adds two penalty terms to CLIP's prompt-learning loss: a Fisher information penalty for covariate shift and a confidence misalignment penalty. The paper reports accuracy and calibration gains, but the combined method is never evaluated as a whole and several claims are internally inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline claim is that the unified loss L_CalShift = Lc + λ1 I(θ) + λ2 CMP works as a single method, but every experiment isolates one penalty (λ2=0 or λ1=0); no result evaluates the combined loss, so the 3.5% accuracy / 5.82% ECE gains are not actually demonstrated for CalShift.","rationale":"The reader's weakest_assumption (Fisher information computability) is serious, and Eq. 1 and Appendix A indeed never give an algorithm for estimating the Hessian in the prompt-learning setting. But even granting a computable Fisher term, the paper would still not support its headline because the combined loss is never run. That is the first-order, load-bearing failure: the object being claimed is not the object being measured. The two single-penalty variants are not a substitute; regularizers can interact, and the paper provides no theorem or experiment ruling out negative interaction. The zero-shot row in Table 1 reinforces this by showing an impossible result under the stated protocol, which suggests the reported numbers may come from a different setup. I therefore keep the reader's REJECT verdict; the recommended path is a combined-loss experiment and a corrected evaluation protocol, not a re-interpretation.","tokens_in":16856,"tokens_out":5081,"duration_ms":47989,"concrete_test":"Run the full CalShift loss, Eq. 3, with both penalties active on ImageNet and the eight covariate-shift datasets, using a small grid such as λ1, λ2 ∈ {0.1, 0.2, 0.4, 0.8} under the same shot settings as Table 1, and report accuracy and ECE for each combined run. Also report the 0-shot protocol explicitly: is any data used, and are prompts updated? If no setting with both λ1 and λ2 nonzero reproduces or improves on the single-penalty results, the unified-claim is falsified; if such a setting exists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive gap is not in the theory but in the experimental design. Eq. 3 defines CalShift as Lc + λ1 I(θ) + λ2 CMP, and the abstract/§1 claim this unified loss yields up to 5.82% ECE reduction and 3.5% accuracy gains. However, every table separates the two regularizers. In Table 1, the accuracy rows compare 'CoOp + FIM' to CoOp (i.e., λ2=0), while the ECE rows compare 'CoOp + CMP' (λ1=0). Tables 2 and 3 have the same split. The ablation study in Appendix B is explicit: 'CoOp + CMP (λ1=0)' and 'CoOp + FIM (λ2=0)'. There is no reported run of Eq. 3 with both λ1 and λ2 nonzero. Thus the headline numbers are for two separate single-penalty methods, not for the unified method being proposed. The assertion in §3 that 'neither penalty degrades the other's effect' is exactly what needs testing: Fisher and CMP regularizers both reshape the loss landscape and can interact, especially in few-shot CLIP where prompt gradients are noisy. Unless a combined run on the same benchmarks matches the isolated gains, the paper's central claim is unsupported. A compounding sign of protocol problems is Table 1's 0-shot row: CoOp+FIM (84.9) cannot differ from CoOp (79.5) if no training examples are available to update the prompts or compute a penalty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CalShift, a unified loss for few-shot classification with CLIP-based prompt learning. The loss is L_CalShift = Lc + λ1I(θ) + λ2CMP, where I(θ) is a Fisher information penalty intended to correct covariate shift and CMP is a confidence misalignment penalty intended to reduce overconfidence. The authors claim this unified objective improves accuracy by up to 3.5% on covariate-shift datasets and reduces Expected Calibration Error by up to 5.82%. Experiments are reported on ImageNet few-shot settings, eleven vision datasets, and eight covariate-shift datasets, always comparing CoOp with either CoOp+FIM or CoOp+CMP against a vanilla CoOp baseline.","tokens_in":17248,"tokens_out":7140,"duration_ms":62625,"significance":"If the unified regularizer actually delivered the claimed simultaneous gains in accuracy and calibration, it would be a practically useful addition to prompt learning for vision-language models. The paper's motivation is reasonable, and the experimental scope is broad: 19 datasets and multiple shot settings are considered, and the ablations separately study each penalty. However, the paper's central claim is not tested: no experiment ever runs the full CalShift objective with both penalties active. Moreover, the zero-shot rows in Table 1 are logically impossible for a training-time regularizer, the hyperparameters are tuned on datasets that also appear in the main tables, and the theoretical propositions are not valid as stated. The strengths are the breadth of the evaluation and the clarity of the single-penalty ablations, but these do not compensate for the absence of evidence for the method as proposed.","major_comments":[{"comment":"The headline claim, stated in the abstract and §1, is that the unified loss L_CalShift = Lc + λ1I(θ) + λ2CMP simultaneously corrects covariate shift and overconfidence. No experiment evaluates this objective. The accuracy rows of Tables 1–3 compare 'CoOp + FIM' to CoOp (i.e., λ2=0), the ECE rows compare 'CoOp + CMP' to CoOp (i.e., λ1=0), and the ablations in Appendix B are explicit that either λ1=0 or λ2=0. The assertion in §3 that 'neither penalty degrades the other's effect' is therefore untested; the reported 5.82% ECE reduction and 3.5% accuracy gain belong to two separate single-penalty methods, not to CalShift. A combined run with both penalties active on the same benchmarks is required to support the paper's title claim.","section":"§4.2, Eq. (3), Tables 1–3"},{"comment":"In the 0-shot column of Table 1, CoOp+FIM reports 84.9% versus 79.5% for CoOp, and CoOp+CMP reports ECE 3.06 versus 3.36. In a genuine zero-shot setting no training examples are available to update prompts or to compute either penalty, so these entries cannot differ from the CoOp baseline. Their difference indicates either a protocol inconsistency (e.g., the penalty computed on some held-out data, or the row not actually zero-shot) or a reporting error. Because the same table is used to claim that FIM 'acts as a prior-preserving regularizer' in zero-shot scenarios, this issue directly undermines the experimental evidence.","section":"Table 1, zero-shot rows"},{"comment":"The hyperparameters λ1 and λ2 are tuned on the same datasets that appear in the main results: Tables 6 and 7 report λ1 and λ2 selection on Flowers102, Food101, UCF101, and DTD, and these four datasets are also included in the main evaluation tables (Tables 2, 4, and 5). The paper does not describe a held-out validation split. The reported improvements are therefore at risk of being artifacts of test-set hyperparameter selection; an independent validation split or a sensitivity analysis that does not reuse evaluation data is needed.","section":"Appendix B, Tables 6–7; §4.1"},{"comment":"The theoretical support is not established as stated. Proposition A.1 claims equality between Fisher information and KL divergence, but the proof only derives a local quadratic approximation DKL ≈ 1/2 (θq−θ)^T I(θ)(θq−θ); no equality is shown, and the assumptions under which the approximation holds are not stated. Proposition A.2's definition of CMP changes between Eq. (2) and the proof (in Eq. (2), y is the true class and y′ the incorrect class, while in the proof y′ is the predicted class), and the limits P(x,y′)→1 and P(x,y′)→0 do not establish the claimed bounds 0≤CMP≤1. In addition, no algorithm is given for computing I(θ) for the CLIP text encoder; Eq. (1) is a definition, not an estimation procedure. Since these propositions are used to justify the loss in Eq. (3), the theoretical argument needs to be corrected or removed.","section":"Appendix A, Propositions A.1 and A.2"},{"comment":"Table 2 contains accuracy degradations that are not acknowledged: CoOp+FIM decreases accuracy on StanfordCars (59.2→54.3, ∆=8.2↓) and EuroSAT (53.8→49.2, ∆=8.6↓). The text says 'improvement across most datasets' but elsewhere claims 'CalShift demonstrates robust performance improvement' and lists only positive cases. The average 3.2% gain is driven by large gains on some datasets; the negative results should be discussed and the robustness claim tempered.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The notation P(x,y) and P(x,y′) is never formally defined; since CLIP operates on image-text pairs, the meaning of x and y in a softmax probability over classes should be stated explicitly.","section":"§3, Eq. (2)"},{"comment":"The introduction and conclusion describe CalShift as 'training-free adaptation', but the method is integrated into CoOp, which learns prompts from training examples; this is contradictory and should be clarified.","section":"§1 and §5"},{"comment":"The blocks in Table 2 are arranged with ECE first and ACC second, while the caption says the upper half is accuracy and the lower half is ECE; the order should be corrected.","section":"Table 2"},{"comment":"The manuscript contains numerous typos and grammatical errors, including 'addresse', 'covriate', 'Calshift', 'confidene', 'emprical', 'ststistics', and 'remains always'; these should be corrected in a careful editing pass.","section":"Throughout"},{"comment":"Proposition 3.1 is not a proposition in the mathematical sense: it restates the intended effect of the two penalties and is followed by no proof; it should be reworded as a design statement rather than a formal claim.","section":"§3, Proposition 3.1"},{"comment":"The paper does not report the number of random seeds, the variance across runs, or the exact few-shot training protocol used for CoOp, which makes the numerical differences in the tables difficult to assess.","section":"§4.1, Implementation details"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is that the unified loss works, but no experiment evaluates the unified loss. The zero-shot FIM and CMP results in Table 1 are internally impossible, which suggests a protocol or reporting error that affects the credibility of the whole experimental section. The close relationship to the authors' earlier works [26, 27] also raises a novelty question: the only new element is the combination, and that combination is never tested. These issues are substantial enough that I cannot recommend revision as a path to acceptance without essentially new experiments and a rewritten theoretical section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: the paper has a plausible idea—add a Fisher-information regularizer for covariate shift and a confidence-misalignment penalty to CoOp prompt tuning—and it tests the pieces across many datasets. But the central claim is not actually verified. Eq. 3 defines CalShift as Lc + λ1 I(θ) + λ2 CMP, yet every experiment uses one penalty at a time. Accuracy is reported with λ2=0, ECE with λ1=0. The headline numbers (5.82% ECE, 3.5% accuracy) are therefore for two different single-penalty methods, not for the unified loss. The combined run is missing.\n\nCredit where due: the paper is honest about borrowing both terms from the authors' prior papers [26] and [27], so the novelty is the combination and the CoOp application. It evaluates on 19 datasets covering standard few-shot and shift benchmarks, and the isolated gains—if reproducible—are the kind of modest, cheap improvements that practitioners care about. That is real, if incremental, value.\n\nThe soft spots are substantial. The zero-shot accuracy row in Table 1 is impossible: CoOp+FIM at 84.9 vs CoOp at 79.5 with no training data to update prompts or compute a Fisher penalty. That is a red flag for the rest of the table. Hyperparameters λ1 and λ2 were tuned on Flowers102, Food101, UCF101, and DTD, all of which appear in the main results, so the reported gains are partly fitted to the test sets. The CMP definition in Eq. 2 and in Appendix A.2 do not match, and the proofs in Appendix A are informal—A.1 proves a local quadratic approximation, not the claimed equality, and A.3 contains a derivative expression that merely states the desired conclusion. No algorithm is given for computing the Fisher Hessian in Eq. 1, which is not trivial for a CLIP text encoder. Also missing: error bars, code, and any analysis of how the two penalties interact, even though the text asserts they don't degrade each other.\n\nOn the theory: the PAC-Bayes and MDL passages are decorative; they are not used to derive anything and could be cut without loss.\n\nWho this is for: someone working on CLIP prompt regularization might want to see the isolated-penalty ablations, but the headline claims should not be cited. The paper deserves a serious referee because the method is testable and the missing experiments are easy to specify. My recommendation: send it to review with a request for major revision—combined-loss runs, a fix to the zero-shot protocol, a consistent CMP, and a proper tuning protocol.","headline":"Plausible regularizer combo for CLIP few-shot tuning, but the paper never tests the combined loss it advertises, and the zero-shot row is impossible.","tokens_in":17761,"tokens_out":3057,"would_cite":false,"duration_ms":26592,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CalShift claims a pair of regularizers added to CLIP's contrastive loss can fix both covariate shift and overconfident predictions in few-shot vision-language classification.","keywords":["few-shot classification","vision-language models","CLIP","covariate shift","confidence calibration","expected calibration error","prompt learning","Fisher information"],"falsifier":"Run the joint loss $L_{\\text{CalShift}} = L_c + \\lambda_1 I(\\theta) + \\lambda_2 \\text{CMP}$ with both $\\lambda_1 > 0$ and $\\lambda_2 > 0$ on the same benchmarks and compare against each penalty alone; the paper never reports this configuration, so if the joint model fails to improve over the isolated penalties on both accuracy and ECE, the central claim of simultaneous correction collapses. A second check is to compute the actual Hessian of the CLIP text encoder's log-likelihood and measure whether minimizing it actually reduces a direct covariate-shift metric such as KL divergence between source and target feature distributions, which the paper's Proposition A.1 only approximates locally.","tokens_in":16597,"feed_emoji":"","tokens_out":6666,"duration_ms":51771,"temperature":0.7,"pith_summary":"CalShift claims that a single regularized loss can fix two problems that plague CLIP-based few-shot classification at once: covariate shift between pre-training and target data, and overconfident wrong predictions. The loss adds a Fisher information penalty to flatten the loss landscape and resist distribution shift, plus a Confidence Misalignment Penalty to push probability mass back toward the true class, inside CoOp prompt learning. Across 19 vision and domain-shift benchmarks, the paper reports that the Fisher penalty improves accuracy by up to 7.2% over CoOp and the confidence penalty reduces Expected Calibration Error by up to 12.5%, with the headline covariate-shift gains being 3.5% accuracy and 5.82% ECE. If these effects hold jointly, CalShift would offer a simple drop-in regularizer for more reliable low-shot vision-language models.","feed_headline":"CalShift: one loss to fight CLIP's covariate shift and overconfidence","feed_subtitle":"Claimed to cut calibration error by up to 5.82% and lift accuracy by 3.5% on shifted benchmarks.","key_machinery":"The machinery is a two-term regularizer added to the contrastive loss: the Fisher information $I(\\theta) = -\\mathbb{E}[\\partial^2 \\log P(X; \\theta)/\\partial \\theta^2]$, the negative Hessian of the log-likelihood, which the paper treats as a curvature penalty that stabilizes the model under distribution shift; and the Confidence Misalignment Penalty $\\text{CMP} = P(x,y)\\sum_{y' \\neq y: P(x,y') > P(x,y)} P(x,y')$, which sums the excess softmax probability assigned to wrong classes that outrank the true class. The combined loss $L_{\\text{CalShift}} = L_c + \\lambda_1 I(\\theta) + \\lambda_2 \\text{CMP}$ is the object that is supposed to carry both corrections at once.","core_discovery":"The paper's central claim is that covariate shift and confidence misalignment are coupled failure modes, and both can be corrected in the prompt-learning setting by minimizing $L_{\\text{CalShift}} = L_c + \\lambda_1 I(\\theta) + \\lambda_2 \\text{CMP}$. The Fisher information term, defined as the negative Hessian of the log-likelihood, is argued to penalize sharp curvature and tighten a PAC-Bayes generalization bound, while the CMP term redistributes softmax likelihood from overconfident incorrect classes to the true class. Empirically, CoOp+FIM beats CoOp on accuracy across all shot counts and most datasets, and CoOp+CMP reduces ECE on nearly all of them; the paper interprets this as evidence that the two penalties are complementary and that their combination improves both robustness and calibration.","pith_inferences":["Editorial inference: Because the paper evaluates FIM and CMP only in isolation, the reported 5.82% ECE and 3.5% accuracy numbers are not evidence that the full $L_{\\text{CalShift}}$ loss with both penalties active reproduces them; a joint experiment is the direct next test.","Editorial inference: The theoretical link between $I(\\theta)$ and covariate shift rests on a local quadratic approximation (Proposition A.1); an empirical check would be to compute both $I(\\theta)$ and a direct shift measure such as KL divergence between source and target feature distributions and see whether they rank shifted benchmarks the same way.","Editorial inference: If the combined regularizer works, it should transfer to other prompt-tuning or adapter methods beyond CoOp, such as class-specific prompts or multi-modal prompt tuning, since neither penalty is specific to CoOp's unified-context design."],"forward_implications":["On ImageNet, adding the Fisher penalty to CoOp raises average accuracy from 79.8% to 84.5% across 0 to 16 shots, with the largest relative gain of 7.2% at 1 shot.","Adding the CMP penalty lowers average ECE on ImageNet from 2.90% to 2.62%, a 9.66% relative reduction, and the improvement grows as shot count increases.","On eight covariate-shift benchmarks (PACS, Office-Home, VLCS, DomainNet, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R), CoOp+FIM improves accuracy on every dataset, up to 3.5%, and CoOp+CMP lowers ECE on every dataset, up to 5.82%.","The Fisher term acts as a prior-preserving regularizer since it also improves zero-shot accuracy over CoOp, suggesting it protects pre-trained knowledge rather than only helping few-shot fitting."],"supporting_citations":[{"why":"Defines the Confidence Misalignment Penalty (CMP) used in the CalShift loss.","marker":"[26]"},{"why":"Source of the Fisher information penalty for covariate shift correction.","marker":"[27]"},{"why":"CLIP, the vision-language model that CalShift regularizes.","marker":"[45]"},{"why":"CoOp prompt learning, the base method that CalShift builds on and compares against.","marker":"[67]"},{"why":"Defines Expected Calibration Error, the main calibration metric used in the experiments.","marker":"[15]"},{"why":"Documents miscalibration of CLIP-based adapters under distribution shift, motivating the need for calibration-aware regularization.","marker":"[37]"},{"why":"Provides the statistical definition of Fisher information as the negative Hessian of the log-likelihood.","marker":"[31]"}],"fun_headline_variants":["CalShift: one loss fixes both covariate shift and overconfidence","Cut CLIP's overconfidence and covariate shift with CalShift","CalShift boosts CLIP few-shot accuracy and calibration","One loss to fix CLIP's overconfidence and drift in few-shot","CalShift: dual penalty corrects CLIP's covariate shift and overconfidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Fisher information $I(\\theta)$ can be computed for the CLIP text encoder and that minimizing it genuinely corrects covariate shift, yet the paper gives no algorithm for computing the Hessian and its appendix proof only establishes a local approximation.","fun_headline_variants_meta":{"raw":{"variants":["CalShift: one loss fixes both covariate shift and overconfidence","Cut CLIP's overconfidence and covariate shift with CalShift","CalShift boosts CLIP few-shot accuracy and calibration","One loss to fix CLIP's overconfidence and drift in few-shot","CalShift: dual penalty corrects CLIP's covariate shift and overconfidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000949,"raw_usage":{"total_tokens":4062,"prompt_tokens":971,"completion_tokens":3091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2999}},"tokens_in":587,"tokens_out":3091,"duration_ms":21601,"temperature":1.0,"reasoning_tokens":2999,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:49:16.585411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the joint loss $L_{\\text{CalShift}} = L_c + \\lambda_1 I(\\theta) + \\lambda_2 \\text{CMP}$ with both $\\lambda_1 > 0$ and $\\lambda_2 > 0$ on the same benchmarks and compare against each penalty alone; the paper never reports this configuration, so if the joint model fails to improve over the isolated penalties on both accuracy and ECE, the central claim of simultaneous correction collapses. A second check is to compute the actual Hessian of the CLIP text encoder's log-likelihood and measure whether minimizing it actually reduces a direct covariate-shift metric such as KL divergence between source and target feature distributions, which the paper's Proposition A.1 only approximates locally.","supporting_citations":[{"cited_title":"Causal covari- ate shift correction using fisher information penalty","cited_arxiv_id":null,"evidence_quote":"Source of the Fisher information penalty for covariate shift correction."},{"cited_title":"Robust calibration of large vision- language adapters","cited_arxiv_id":null,"evidence_quote":"Documents miscalibration of CLIP-based adapters under distribution shift, motivating the need for calibration-aware regularization."},{"cited_title":"Theory of point esti- mation","cited_arxiv_id":null,"evidence_quote":"Provides the statistical definition of Fisher information as the negative Hessian of the log-likelihood."}],"review_version":1}