{"id":"d50c7fa0-e512-4b5b-a5e5-e90c4b63bcca","arxiv_id":"2512.12997","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adversarial fine-tuning of CLIP with Dirichlet-distribution alignment improves zero-shot robustness and restores a sensible ordering of predictive uncertainty.","lead":"This paper adds a fine-tuning step to CLIP that treats its class scores as a Dirichlet distribution, then aligns the distribution of attacked images with that of clean ones. The goal is to fix a reliability gap where adversarial attacks make CLIP both wrong and overconfident.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix E.1's claim that softplus and τ-switching preserve AU/EU ordering is unproven and likely false; ranking-based calibration metrics are therefore unreliable.","rationale":"The reader's weakest_assumption exactly matches the most load-bearing concern I can identify. The central claim is that UCAT improves both adversarial robustness and uncertainty calibration. The robustness part (Table 2, Table 5) is plausible, though gains are modest. The calibration part, however, is measured primarily through AUROC of AU and EU, which are ranking-based metrics. Appendix E.1 admits to changing the uncertainty definition from the theory (exp map) to softplus and to using different temperatures for AU and EU, justifying this by asserting that ordering is preserved. This assertion is not proven and is mathematically suspect, because AU is a ratio-dependent nonlinear function of the α vector and EU depends monotonically on α0, which itself is a nonlinear function of τ. The empirical results in Fig. 3b are therefore vulnerable to an implementation-specific artifact: the reported AU/EU improvements may be an artifact of the chosen τ/softplus rather than of the UCAT loss. The paper's limitation statement does not acknowledge this. A concrete check—comparing rank correlations and AUROC under the two mappings—would settle whether the concern lands. My recommendation is UNCHANGED because the reader already set CONDITIONAL; this stress test reinforces that conditionality without moving the verdict to reject, since the robustness results remain valuable.","tokens_in":27798,"tokens_out":9828,"duration_ms":88523,"concrete_test":"Reproduce AU/EU computation on a fixed set (e.g., 1000 images per dataset from the 16 benchmarks, both clean and PGD-adversarial) with the official mapping (softplus, τ_AU=0.01, τ_EU=0.07) and with the theoretical mapping (α=exp((τℓ+1)/τ'), single τ=0.01 for both AU and EU). Compute Spearman rank correlations between the AU values from the two mappings and between the EU values, and compute AUROC for the adversarial-vs-clean detection task under both mappings. If the rank correlation is below ~0.95, or if the AUROC differs by more than a few points, the Appendix E.1 ordering assertion fails, and the reported calibration results are not robust to implementation choices.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's calibration claim rests on AU-AUROC and EU-AUROC (Fig. 3b), which depend only on the ranking of uncertainty scores. Appendix E.1 introduces two ad hoc modifications: h'(ℓ)=softplus(h(ℓ)) and using τ=0.01 for AU but τ=0.07 for EU, asserting they 'affect only the magnitude of uncertainty values, not their ordering.' This is not established and is questionable. AU is the expected entropy of a Dirichlet with concentration vector α; it is a nonlinear function of the entire α vector, not a monotone transform of a scalar. Scaling all α components by a constant changes AU in a way that depends on α0 and the relative proportions, so the ordering between two samples can invert. Similarly, EU = C/(α0+C) is monotone in α0, but α0 = Σ exp(softplus((τℓ_k+1)/τ')) depends nonlinearly on τ; changing τ from 0.01 to 0.07 alters α0 differently across samples, so EU rankings may also change. Moreover, with the theoretical mapping and τ'=0.07, α0 is astronomically large (~e^{28}), making EU effectively 0 for all samples; the τ=0.07 switch is what creates any EU variance, meaning the reported EU-AUROC is an artifact of the chosen temperature, not a faithful reflection of the model's uncertainty. Without a proof or empirical verification of order preservation, the AU/EU AUROC improvements cannot be attributed to UCAT. The paper provides no such proof; the claim 'theoretically acceptable' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UCAT, an adversarial fine-tuning objective for zero-shot CLIP. It reparameterizes CLIP logits as Dirichlet concentration parameters α_k = exp((τℓ_k+1)/τ′), defines closed-form aleatoric and epistemic uncertainties from the induced Dirichlet distribution, and regularizes training by aligning the adversarial Dirichlet with the clean one via KL divergence in addition to a text-guided cross-entropy term. The method is evaluated on 16 single-label datasets and MS-COCO, under PGD, CW, AutoAttack, CAA, and A3 attacks, comparing against TeCoA, FARE, PMG-AFT, and TGA-ZSR. The paper reports improved average adversarial robustness, competitive clean accuracy, and improved calibration measured by ECE and AU/EU AUROC.","tokens_in":28159,"tokens_out":10290,"duration_ms":101708,"significance":"If the results hold, the paper makes a useful empirical contribution: it highlights the over-confidence of CLIP under adversarial perturbations and shows that a simple Dirichlet-level distributional regularizer can improve robustness and calibration across many zero-shot benchmarks. The experimental coverage is broad (16 datasets, multiple attack protocols, stronger training regimes, several backbones, multi-label evaluation), and the ablation study isolates the benefit of aligning Dirichlet parameters over softmax probabilities. However, the calibration claims currently rest on an unproven and questionable ordering-preservation assertion in Appendix E.1, and the main hyperparameters are selected on the same 16-dataset benchmark used for evaluation. These issues prevent the paper from being accepted in its current form.","major_comments":[{"comment":"The assertion that 'both the softplus mapping and the rescaling by τ affect only the magnitude of uncertainty values, not their ordering' is load-bearing for the AU/EU AUROC results in Fig. 3b, but it is not proven and is not implied by monotonicity of the activation. AU in Eq. (6) is a nonlinear function of the whole α vector; EU in Eq. (7) is monotone in α0, but α0 changes nonlinearly under the τ-switch and softplus in sample-dependent ways. Moreover, with the theoretical mapping at τ′=0.07, α0 is astronomically large (≈ exp(28)), so EU ≈ 0 for all samples; the τ=0.07 switch is what creates any EU variance, making the reported EU-AUROC an artifact of the chosen evaluation temperature. Since AUROC is rank-based, even pairwise ranking inversions change the metric. Please provide a formal proof or an empirical verification of rank preservation on the actual feature distributions, or remov","section":"Appendix E.1 / Fig. 3b"},{"comment":"The calibration coefficient τ′ and the regularization weight λ are selected on the same 16 datasets used to report the headline results. Table 7 sweeps τ′ over {0.01,…,0.10} and chooses 0.07 by the average harmonic mean on the 16 test datasets; Fig. 3a selects λ·β by the same average. This is selection on the test benchmark, so the reported averages are optimistic and the margin over baselines is not a fair estimate of generalization. Use a validation protocol that does not touch the evaluation datasets (e.g., select on TinyImageNet validation or a held-out subset of the 16 datasets), or report results for all swept values with clear disclosure that the reported numbers are the best on the test set.","section":"Sec. 6.3 / Appendix F.3 / Table 7 / Fig. 3a"},{"comment":"All results appear to be single runs with no error bars or significance testing, while several headline differences are very small (e.g., Table 2 AutoAttack average: 30.58 vs. 30.52 for TGA-ZSR; Table 5 AutoAttack average: 20.67 vs. 19.98; Table 1 mAP: 37.60 vs. 38.23 for TGA-ZSR). Without variance estimates over multiple seeds or bootstrap confidence intervals, the reader cannot assess whether the claimed improvements are meaningful. Please add at least 3 seeds and report mean±std for the main tables and figures, or justify why single-run comparisons are sufficient in this setting.","section":"All experiments (Tables 2, 5, 6, 8)"},{"comment":"The paper describes the Dirichlet reformulation as 'a theoretically justified and closed-form approach,' but Lemma 4.3 is essentially a softmax identity: defining α=exp(h(ℓ)) gives p_Dir=softmax(sℓ) by construction. This does not establish that CLIP's contrastive objective induces a Dirichlet distribution over class probabilities, nor that CLIP logits are 'evidence' in the EDL sense. The reformulation is a modeling choice whose utility is empirical; the theoretical framing overstates the result. I recommend tempering the claims and clearly distinguishing the reparameterization identity from an actual probabilistic interpretation of CLIP's training objective.","section":"Section 4 / Abstract"}],"minor_comments":[{"comment":"Typo: 'naturally admits provides closed-form uncertainty decomposition' should be 'naturally provides a closed-form uncertainty decomposition'.","section":"Section 4"},{"comment":"Typos: 'accuray invariance' should be 'accuracy invariance'; 'calibaration control' should be 'calibration control'.","section":"Appendix D.2.1"},{"comment":"The symbol τ is overloaded: it denotes the CLIP temperature in Eq. (8) and also the ad hoc evaluation temperature in Appendix E.1. Please clarify the notation, especially whether τ in the uncertainty computation is the model's learned temperature or a free evaluation parameter.","section":"Notation"},{"comment":"The reproducibility statement says only that implementation details are in the appendix; no code is provided. Given the number of implementation choices (τ′, λ, softplus, temperature switch), releasing code would greatly improve reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The empirical robustness comparison is reasonably thorough, but the calibration analysis needs serious revision: the ordering-preservation claim in Appendix E.1 is unsupported and likely false, and the hyperparameter selection on the evaluation benchmark is a validity concern. If the authors can validate rank preservation (or remove the AU/EU AUROC claims), move hyperparameter selection off the test datasets, and add variance estimates, the paper could become publishable. I see no indication of misconduct, but the current version overstates the theoretical and calibration contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"UCAT is a new objective for adversarial fine-tuning of CLIP that aligns Dirichlet distributions derived from clean and adversarial logits. The robustness gains look real; the calibration claims are undermined by an unproven implementation choice.\n\nThe genuinely new bit is the loss L = L_ce + λ KL(Dir(α_adv) ∥ Dir(α)). The Dirichlet reformulation itself is not new—as the paper's own Corollary 4.3.1 states, it is just softmax with temperature s=τ/τ'. Calling it a 'theoretical account' of CLIP uncertainty overstates the case; the equivalence is a tautology for any logits. What is useful is the empirical finding that aligning the full concentration vectors outperforms aligning softmax probabilities (Table 3).\n\nThe main robustness results across 16 datasets seem solid enough. The method matches or beats TeCoA, FARE, PMG-AFT, and TGA-ZSR in several settings, with modest gains in the main PGD/AutoAttack averages. The MS-COCO results also look reasonable.\n\nThe load-bearing problem is the uncertainty evaluation. Appendix E.1 switches to softplus and, crucially, uses τ=0.01 for AU and τ=0.07 for EU, asserting this affects only magnitudes, not rankings. That assertion is unproven and likely wrong. AU is a nonlinear function of the whole α vector, and changing τ rescales α unevenly across samples, so sample-wise AU and EU orderings can invert. The AU-AUROC and EU-AUROC numbers in Fig. 3b depend entirely on those rankings. Without a proof or an empirical check that rankings are preserved, the calibration improvements are not established. This is not a minor footnote; it is the paper's second headline claim.\n\nThere are also the usual concerns: τ' and λ are chosen on the same 16-dataset benchmark used for reporting, and there are no error bars. These inflate confidence in the numbers but are common in this subfield.\n\nBottom line: the robustness method is worth trying and the paper deserves a serious referee, but the calibration section has to be repaired—either prove order preservation, compute AU/EU with the actual τ, or use metrics that are invariant to monotone transforms. I'd like to see this as a workshop or conference paper after those fixes.","headline":"Plausible new robustness loss for adversarial CLIP; the calibration claims rest on an unproven ordering assumption that likely fails.","tokens_in":28677,"tokens_out":4291,"would_cite":true,"duration_ms":42669,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UCAT restores calibrated uncertainty in adversarial CLIP by aligning Dirichlet distributions of clean and attacked inputs","keywords":["CLIP","zero-shot adversarial robustness","uncertainty calibration","Dirichlet distribution","evidential deep learning","aleatoric uncertainty","epistemic uncertainty","adversarial fine-tuning"],"falsifier":"Across the 16 evaluation datasets, compute the per-sample ranking of AU/EU twice: once with α=exp((τℓ+1)/τ′) and once with the softplus-based α and the different τ values used in reporting; if the Kendall rank correlation drops well below 1 on any dataset, the ordering-preservation premise fails and the calibration numbers need to be recomputed under one consistent mapping.","tokens_in":27668,"feed_emoji":"🛡️","tokens_out":5570,"duration_ms":55913,"temperature":0.7,"pith_summary":"Adversarial perturbations do more than lower CLIP's accuracy: they often make the model more confident, producing low predictive uncertainty on inputs that are actually hard. The paper argues that existing zero-shot adversarial fine-tuning methods overlook this miscalibration because they align only the winning logit (or single text anchor) and ignore the geometry of nearby classes. It proposes UCAT, which re-reads CLIP's logits as the concentration parameters of a Dirichlet distribution and then regularizes the entire clean and adversarial Dirichlet distributions against each other with a KL term. On 16 single-label datasets and MS-COCO, the paper reports that this restores a sensible ordering of uncertainty (clean original < fine-tuned clean < fine-tuned adversarial), improves adversarial robustness relative to prior single-anchor methods, and preserves clean accuracy. The paper itself scopes these claims to image-encoder attacks, to fine-tuned (not test-time) settings, and to CLIP-family backbones.","feed_headline":"One KL term restores CLIP's uncertainty under attack","feed_subtitle":"A Dirichlet reparameterization turns CLIP logits into evidence, fixing adversarial over-confidence and preserving clean accuracy.","key_machinery":"The Dirichlet reparameterization α_k(x) = exp((τℓ_k(x)+1)/τ′) of CLIP logits into concentration parameters; the paper proves this is a valid evidence mapping (α_k≥1, monotone) and that it equals CLIP's softmax exactly when τ′=τ, with s=τ/τ′ acting as a temperature that sharpens or flattens predictions without changing the argmax. The training mechanism is the KL divergence KL(Dir(α_adv)∥Dir(α)) between the attacked and clean Dirichlet distributions, combined with the text-guided cross-entropy term; this is what carries the uncertainty calibration by matching both AU (class-ambiguity shape) and EU (total evidence α0) simultaneously.","core_discovery":"The central discovery is a structural correspondence: CLIP's softmax over image–text logits is exactly the mean of a Dirichlet distribution whose concentration parameters are α_k = exp((τℓ_k+1)/τ′). Choosing τ′=τ recovers CLIP's predictive distribution verbatim, so CLIP's pretrained logits can be treated as evidence without changing the model's predictions. This gives closed-form aleatoric and epistemic uncertainty estimates from one forward pass. The paper's method, UCAT, fine-tunes the image encoder with L = L_ce + λ KL(Dir(α_adv) ∥ Dir(α)), aligning the attacked sample's evidence distribution back to the clean one instead of pulling a single logit to the ground-truth anchor. The paper cla","pith_inferences":["A natural next test the authors do not run: use EU = C/(α0+C) as a per-sample adversarial detector. Since UCAT raises EU specifically on attacked inputs, thresholding EU could flag perturbations in-the-wild without a separate detector.","The same Dirichlet evidence view could be applied to text-encoder attacks or to prompt-ensemble diversity; the paper confines itself to image perturbations, but the concentration-parameter identity is agnostic to which encoder produced the logits.","If the ordering-preservation assumption in Appendix E.1 (softplus and different τ's leave ranking unchanged) is violated, AUROC comparisons would need recalculation with a single consistent mapping; a direct rank-correlation check is the cheap way to settle it.","The temperature ratio s=τ/τ′ (<1) gives a free calibration dial on clean inputs too; one could tune s on a validation set to lower ECE without retraining."],"forward_implications":["Attacked inputs can be recognized by their uncertainty signature: after UCAT fine-tuning, adversarial images no longer look spuriously confident, so uncertainty itself becomes a usable reliability signal under perturbation.","Because KL(Dir(α_adv)∥Dir(α)) aligns the whole evidence vector, the method preserves relative inter-class geometry, which the paper argues is the reason clean accuracy and zero-shot transfer are retained while robustness improves.","Closed-form AU and EU estimates mean reliability can be monitored per sample with no sampling or extra model, a property inherited directly from the Dirichlet reparameterization.","The same reparameterization works across the CLIP-family backbones the paper tests, suggesting the correction is not tied to one architecture."],"fun_headline_variants":["Turn CLIP logits into evidence, beat adversarial over-confidence","Dirichlet reparameterization fixes CLIP's adversarial over-confidence","Align CLIP's evidence to restore calibrated uncertainty under attack","One Dirichlet KL aligns CLIP's evidence, fixing adversarial over-confidence"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's calibration results depend on the assumption in Appendix E.1 that switching from the exponential evidence map to softplus and using τ=0.01 for aleatoric versus τ=0.07 for epistemic uncertainty changes only the magnitude of uncertainty values and never their ordering; if that ordering shifts, the reported AUROC improvements would not measure what they claim.","fun_headline_variants_meta":{"raw":{"variants":["Turn CLIP logits into evidence, beat adversarial over-confidence","Dirichlet reparameterization fixes CLIP's adversarial over-confidence","Align CLIP's evidence to restore calibrated uncertainty under attack","One Dirichlet KL aligns CLIP's evidence, fixing adversarial over-confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3461,"prompt_tokens":726,"completion_tokens":2735,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2663}},"tokens_in":470,"tokens_out":2735,"duration_ms":20894,"temperature":1.0,"reasoning_tokens":2663,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T16:30:19.322914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Across the 16 evaluation datasets, compute the per-sample ranking of AU/EU twice: once with α=exp((τℓ+1)/τ′) and once with the softplus-based α and the different τ values used in reporting; if the Kendall rank correlation drops well below 1 on any dataset, the ordering-preservation premise fails and the calibration numbers need to be recomputed under one consistent mapping.","supporting_citations":[],"review_version":1}