{"id":"8d4f584f-494a-41b8-9d27-fb0f24eb4600","arxiv_id":"2510.02913","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CAW adds a confidence-weighted KL loss and feature-alignment regularization to CLIP adversarial fine-tuning, raising average AutoAttack robust accuracy from 31.6% to 33.5% on 15 datasets.","lead":"The authors propose a new training loss, Confidence-Aware Weighting (CAW), that makes CLIP image classifiers more resistant to adversarial attacks by focusing training on the most easily fooled images. It improves robust accuracy on average across 15 datasets compared with previous state-of-the-art methods, while using less memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison likely unfair: CAW's gains over PMG-AFT/TGA-ZSR rely on copied baseline numbers with unspecified epochs, prompts, and test-set-tuned hyperparameters.","rationale":"The reader's weakest assumption correctly identifies that the reported baseline numbers are copied from prior papers without a fully specified shared protocol. My stress-test corroborates this and adds two concrete observations: (1) the paper explicitly admits no error bars (checklist item 7), so the 1.88% average gain over TGA-ZSR may not be statistically significant; (2) at a larger perturbation bound (4/255), the advantage over PMG-AFT nearly vanishes, indicating the result is sensitive to evaluation details. These concerns do not disprove the method—the ablation in Table 6 shows each loss component helps, and the method is plausible—but they do prevent acceptance as a strong empirical claim. The paper should be conditional pending a fair, well-specified comparison. Since the reader's verdict is already CONDITIONAL, my read does not change it.","tokens_in":15957,"tokens_out":7431,"duration_ms":58436,"concrete_test":"Obtain the official implementations of PMG-AFT and TGA-ZSR, train them on TinyImageNet with the exact same number of epochs, learning-rate schedule, prompt templates, and data preprocessing as used for CAW (to be disclosed by the authors), and evaluate all methods on the 15 datasets under AutoAttack at epsilon=1/255. If the average CAW-minus-best-baseline gap is less than 1% or reverses, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (CAW outperforms PMG-AFT and TGA-ZSR under AutoAttack, Table 1) rests entirely on baseline numbers taken from prior publications (Appendix C states 'we compare against the reported results ... as presented in the TGA-ZSR paper'). However, Appendix B gives only lr=1e-4, momentum=0.9, batch=128, and PGD-2 with epsilon=1/255; it does not state the number of training epochs, learning-rate schedule, prompt templates, or exact evaluation protocol. If CAW was trained for more epochs or used different prompt templates, the reported 2% average gain (33.51% vs 31.63% for TGA-ZSR) could be an artifact of the comparison rather than the method itself. Moreover, α=6 and β=3 are set 'to balance clean and robust accuracy' without a held-out validation protocol, so the reported numbers may be optimistically tuned to the test set. Independent evidence of fragility: at epsilon=4/255 (Table 5), CAW's advantage over PMG-AFT shrinks to 0.26% average (19.53% vs 19.27%), and CAW is worse on several datasets, suggesting the headline improvement is specific to the small perturbation bound used in Table 1. Without matching training budgets, prompt templates, and validation protocols, the claimed superiority is not convincingly established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Confidence-Aware Weighting (CAW), an adversarial fine-tuning method for zero-shot CLIP. CAW adds two terms to the standard cross-entropy loss: (1) a confidence-aware KL divergence between the frozen model's clean-image predictions and the fine-tuned model's adversarial-image predictions, weighted by (1 - P_adv,y), so that uncertain samples receive larger gradients; and (2) an L2 feature alignment regularizer between frozen and fine-tuned image encoder features on adversarial inputs. The model is a ViT-B/32 CLIP fine-tuned on TinyImageNet with PGD-2 at epsilon=1/255 and evaluated on 15 datasets under AutoAttack, PGD-100, and CW. The central claim is that CAW outperforms recent zero-shot robustness methods PMG-AFT and TGA-ZSR in robust accuracy while using less memory. Tables 1, 2, 4, and 5 report average robust accuracy improvements; Table 7 reports memory usage and training time. The appendices include ablations (Table 6) and the NeurIPS checklist, which explicitly states that no error bars or significance tests are reported.","tokens_in":16315,"tokens_out":4922,"duration_ms":40825,"significance":"If the empirical claims are correct, CAW is a simple and intuitive contribution to zero-shot adversarial robustness: it reweights loss terms toward uncertain adversarial samples and uses feature alignment to preserve pre-trained knowledge. The paper evaluates on a broad set of 15 datasets and reports both clean and robust accuracy, which is valuable. The memory-efficiency comparison (Table 7) is also a useful practical consideration. However, the significance of the claimed state-of-the-art results is currently conditional on the validity of the baseline comparison and on the statistical reliability of the 2-point average improvements. The authors are transparent about the lack of error bars, but this transparency does not remedy the fact that the central claim rests on uncontrolled comparisons and hand-tuned hyperparameters. The method itself is plausible, and the paper has merit as a workshop contribution, but the evidence as presented does not yet convincingly establish the headline claim.","major_comments":[{"comment":"The state-of-the-art claim relies on baseline numbers copied from the TGA-ZSR paper ('we compare against the reported results ... as presented in the TGA-ZSR paper'). Appendix B reports only learning rate, momentum, weight decay, batch size, PGD iterations, and epsilon; it does not specify the number of training epochs, learning-rate schedule, prompt templates, or the exact evaluation protocol for CAW or the baselines. Since the reported average gain over TGA-ZSR is 1.9 points (33.51 vs 31.63) and over PMG-AFT is 2.0 points, the comparison is not controlled. Please re-run the baselines under identical training budgets and evaluation conditions, or provide evidence that the copied numbers were produced under exactly the same protocol.","section":"Appendix C / Table 1"},{"comment":"Hyperparameters alpha=6 and beta=3 are chosen 'to balance clean and robust accuracy' without a held-out validation split. Because the same 15 datasets are used for both model selection and final reporting, the reported numbers may reflect optimistic bias. Please provide a sensitivity analysis over alpha and beta and a validation-based selection procedure (e.g., TinyImageNet val), and report the average and standard deviation over multiple seeds. The NeurIPS checklist (Question 7) explicitly states that no error bars or significance tests are reported, but the main claim is an average 2-point improvement; without variance estimates, it is impossible to tell whether the difference is meaningful.","section":"Appendix B / Table 6"},{"comment":"The claim of 'state-of-the-art under strong adversarial attacks' is mostly established at epsilon=1/255. At epsilon=4/255 the average advantage over PMG-AFT shrinks to 0.26 points (19.53 vs 19.27), and CAW is worse on SUN397, OxfordPets, EuroSAT, Caltech-101, and PCAM. This fragility suggests the benefit is specific to a small perturbation bound. Please either soften the claim to the evaluated setting or analyze why the method degrades at larger epsilon.","section":"Table 5"},{"comment":"The memory-efficiency claim is load-bearing for the abstract, but the paper does not describe how memory usage was measured (e.g., whether it includes activations, optimizer states, gradient checkpointing, or is simply the peak GPU allocation reported by PyTorch). If the baseline memory numbers are taken from prior papers with a different measurement protocol, the comparison is not valid. Please specify the measurement methodology and, ideally, report memory using a standard profiler for all methods under the same batch size and resolution.","section":"Appendix B / Table 7"}],"minor_comments":[{"comment":"The summation in Eq. (9) runs from i=0 to N, while Eq. (8) uses i=1 to N. The indexing should be made uniform and consistent with the batch definition.","section":"Eq. (9)"},{"comment":"Several entries are missing spaces, e.g., '18.5419.95', '7.5748.84', making the table hard to read. Please format the table properly.","section":"Table 1"},{"comment":"Reference [1] is cited as 'Language models are few-shot learners' but the author list is incorrect: it lists Ben Mann et al. rather than the GPT-3 authors (Tom B. Brown et al.). Please correct the citation.","section":"References"},{"comment":"The caption reads 'Zero-shot clean accuracy under PGD-100'. Clean accuracy is not measured under an attack; the caption should read 'Zero-shot clean accuracy' or 'clean accuracy (no attack)'.","section":"Table 3"},{"comment":"There is a typo: 'Broder impact' should be 'Broader impact'.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The core idea is simple and potentially useful, but the empirical evidence as presented does not support the state-of-the-art claim because the comparison is not controlled. The paper could become acceptable if the authors re-run the baselines under identical conditions, add error bars and a validation-based hyperparameter selection, and qualify the claims to the evaluated setting. The incorrect attribution of the GPT-3 reference should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The method is a sensible incremental idea: scale a forward KL between adversarial and frozen clean predictions by the model's uncertainty, and add an L2 feature alignment to the frozen encoder. That exact combination is new, and the ablation in Table 6 shows each component helps. The memory comparison in Table 7 is also a useful data point. The paper is clearly written and honest enough to admit in the checklist that no error bars are reported.\n\nBut the central claim — that CAW outperforms PMG-AFT and TGA-ZSR under AutoAttack — is fragile. The baselines are taken from the TGA-ZSR paper, not re-run under the same protocol. Appendix B lists only learning rate, momentum, batch size, and PGD-2 with epsilon=1/255; it doesn't say how many epochs, what prompt templates, or the exact evaluation pipeline. The authors say they adopt settings consistent with prior studies, but that's not enough to rule out a training-budget mismatch. On top of that, alpha=6 and beta=3 are chosen \"to balance clean and robust accuracy,\" which is tuning on the benchmark itself. That makes the reported numbers optimistic.\n\nThe fragility shows up at larger perturbations: at epsilon=4/255 under PGD-100, the average advantage over PMG-AFT shrinks to 0.26 points and CAW is worse on several datasets. So the headline result at 1/255 may reflect a specific benefit for small perturbations rather than a general robustness gain.\n\nWho should read it: people working on zero-shot adversarial robustness of vision-language models, as a quick look at a reasonable confidence-weighting variant. It's not a must-read, but it's a fair workshop-level empirical paper. A serious referee should send it back asking for re-run baselines with matched settings, a held-out validation protocol for hyperparameters, and error bars. The core idea is plausible; the evidence as presented is suggestive, not conclusive.\n\nRecommendation: don't desk-reject. Send it to peer review with those requests.","headline":"A plausible confidence-weighting loss for CLIP robustness, but the reported gains over prior methods are not convincingly established because the comparison uses copied baseline numbers and test-set-tuned hyperparameters.","tokens_in":16781,"tokens_out":2543,"would_cite":false,"duration_ms":79215,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that emphasizing uncertain adversarial examples during fine-tuning—scaling the KL divergence between clean and adversarial predictions by the model's uncertainty—raises CLIP's average robust accuracy under AutoAttack to 33","keywords":["adversarial robustness","zero-shot learning","CLIP","vision-language models","confidence-aware weighting","feature alignment regularization","AutoAttack","adversarial fine-tuning"],"falsifier":"Re-run CAW and the two baselines with identical epochs, optimizer schedules, prompt templates, and multiple seeds, then report AutoAttack averages with error bars; if CAW no longer surpasses the baselines, the central claim fails. Alternatively, ablate the confidence factor by setting 1 - P_adv_{i,y_i} to a constant: if robust accuracy does not drop, the weighting is not what drives the gain.","tokens_in":15864,"feed_emoji":"🛡️","tokens_out":3975,"duration_ms":30053,"temperature":0.7,"pith_summary":"The paper proposes Confidence-Aware Weighting (CAW), a loss for adversarially fine-tuning CLIP without labeled data beyond TinyImageNet. Its central claim is that training should spend more effort on adversarial examples the model is unsure about: it scales the KL divergence between adversarial and clean predictions by one minus the model's confidence in the true class. Added to the standard cross-entropy loss and an L2 feature-alignment regularizer, this yields 33.51% average robust accuracy across 15 zero-shot datasets under AutoAttack, ahead of recent methods PMG-AFT and TGA-ZSR while using less training memory. A sympathetic reader would take the contribution as evidence that uncertainty-weighted losses are a useful, cheap addition to zero-shot adversarial fine-tuning.","feed_headline":"Confidence-aware loss lifts CLIP zero-shot robustness to 33.5%","feed_subtitle":"A confidence-aware KL term beats prior zero-shot defenses on 15 datasets while using less training memory.","key_machinery":"The carrying mechanism is the confidence-aware KL term. It computes the KL divergence from the fine-tuned model's adversarial prediction distribution to the frozen model's clean prediction distribution, then multiplies by 1 - P_adv_{i,y_i}, the model's uncertainty about the true class on the adversarial input. This makes the gradient concentrate on samples that are genuinely unstable under perturbation. A second mechanism, feature alignment regularization, computes the L2 distance between frozen and fine-tuned image-encoder features on adversarial inputs, acting as an anchor that preserves pre-trained semantics and counteracts forgetting during fine-tuning.","core_discovery":"CAW's central claim is that not all adversarial examples deserve equal weight during fine-tuning. The paper defines a Confidence-Aware loss, L_CA = (1/N) sum_i KL(P_adv || P_clean) * (1 - P_adv_{i,y_i}), where P_adv is the fine-tuned model's softmax distribution on an adversarial image, P_clean is the frozen CLIP model's distribution on the clean image, and the factor down-weights examples already predicted confidently while up-weighting uncertain ones. Combined with a feature-alignment term that pulls fine-tuned adversarial image features toward frozen CLIP features via L2 distance, the total loss L_CE + alpha*L_CA + beta*L_Reg is trained with PGD-2 on TinyImageNet. The paper reports that t","pith_inferences":["A natural extension is replacing the scalar confidence weight with a learned or attention-based weight; the paper itself names attention as future work, so this is an invited direction.","Because alpha and beta are fixed without a held-out validation protocol, the claim would be stronger if the gains persist under hyperparameters chosen on a validation split or through a small sweep.","The same uncertainty-weighting principle could apply to the text encoder or to other vision-language models, which the paper notes it did not test.","The observed clean-accuracy improvement suggests the weighting may act as a regularizer, not just a robustness term; this could be tested by measuring feature-space alignment on clean inputs after training."],"forward_implications":["If CAW is correct, zero-shot CLIP can be hardened against strong attacks like AutoAttack with a simple loss modification, without adding parameters or extra data.","The method improves both clean and robust accuracy over PMG-AFT under PGD-100 and CW, suggesting the uncertainty weighting reduces the usual robustness-accuracy trade-off in this setting.","The lower memory footprint makes the approach feasible on a single consumer GPU, which matters for large-scale vision-language fine-tuning.","The gains transfer across 15 datasets spanning objects, scenes, textures, satellite imagery, and medical images, implying the learned robust features generalize beyond the fine-tuning domain.","The choice of KL direction—adversarial distribution as the first argument—is a reusable design decision for other robust fine-tuning objectives."],"fun_headline_variants":["Confidence-aware weighting lifts CLIP robust accuracy to 33.5%","Down-weight confident examples to harden CLIP against adversarial attacks","Uncertain samples get extra weight in new CLIP defense CAW","Weighting uncertain adversarial examples improves CLIP robustness","CAW fine-tunes CLIP by focusing on low-confidence adversarial samples"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that CAW's reported advantage over PMG-AFT and TGA-ZSR is not an artifact of unstated training details, since the paper adopts 'settings consistent with prior studies' without specifying epochs, learning-rate schedule, or prompt templates, and fixes alpha and beta without a held-out validation protocol.","fun_headline_variants_meta":{"raw":{"variants":["Confidence-aware weighting lifts CLIP robust accuracy to 33.5%","Down-weight confident examples to harden CLIP against adversarial attacks","Uncertain samples get extra weight in new CLIP defense CAW","Weighting uncertain adversarial examples improves CLIP robustness","CAW fine-tunes CLIP by focusing on low-confidence adversarial samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3005,"prompt_tokens":752,"completion_tokens":2253,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2164}},"tokens_in":496,"tokens_out":2253,"duration_ms":12707,"temperature":1.0,"reasoning_tokens":2164,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T12:37:54.659901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CAW and the two baselines with identical epochs, optimizer schedules, prompt templates, and multiple seeds, then report AutoAttack averages with error bars; if CAW no longer surpasses the baselines, the central claim fails. Alternatively, ablate the confidence factor by setting 1 - P_adv_{i,y_i} to a constant: if robust accuracy does not drop, the weighting is not what drives the gain.","supporting_citations":[],"review_version":1}