{"id":"7480a9d6-d0df-4ad1-9a89-cc7e6aed57d4","arxiv_id":"2505.13232","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"StarFT improves the OOD robustness of fine-tuned zero-shot models by regularizing the output distribution on LLM-generated spurious textual descriptions to match the frozen zero-shot model.","lead":"StarFT is a fine-tuning method for CLIP-like models that adds a regularization term to stop the model from leaning on spurious cues such as background or texture. It aligns the model's predictions on LLM-generated 'spurious' captions with the original zero-shot model, and the authors report gains on domain-shift, group-shift, and zero-shot benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The spurious-specific gain over random suffix (Table 6) is only 0.4% OOD average with no error bars; without seeded replication or a Lipsum-FT comparison, the central 'spuriosity alignment' mechanism is not distinguishable from generic random-text regularization.","rationale":"The reader's weakest assumption concerned adequacy and unbiasedness of the LM-generated spurious descriptor bank. My concern is adjacent but sharper: the paper's own random-suffix ablation (Table 6) shows that most of the benefit comes from adding any suffix, and the spurious-specific increment is only 0.4% OOD average on a single run. Because L_Star is a KL agreement with a zero-shot reference that itself has known spurious biases, the mechanism cannot be inferred from the method's design alone; it must be established that the semantic content of the descriptors matters. A seeded random-vs-spurious comparison and a direct Lipsum-FT baseline would settle this. The reader's CONDITIONAL verdict already flags missing baselines and error bars, so the verdict does not need to change; the condition for acceptance should be made explicit as this control.","tokens_in":18643,"tokens_out":7785,"duration_ms":82440,"concrete_test":"Re-run the ImageNet ViT-B/16 protocol from Table 6 with K=5 seeds for (1) StarFT with the exact background+texture+resolution bank, (2) the same L_Star loss with the spurious suffixes replaced by random word suffixes matched for length and position, and (3) Lipsum-FT under the same schedule and batch size. Report mean and std of ImageNet ID and OOD Avg, plus Waterbirds WG. If the spurious-minus-random OOD gap is within one standard error, or Lipsum-FT is within noise of StarFT, the spuriosity-specific component is not the driver of the reported gains and the central claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (3) minimizes DKL(~q_i || q_i), so L_Star is an agreement loss with a reference zero-shot model that itself carries spurious biases (Waterbirds zero-shot WG 25.9 vs Avg 87.1), not a direct penalty on spurious features. The only place spuriousness enters is the choice of captions S. The central claim therefore requires that spurious-suffix captions be substantially more effective than random-suffix captions. Table 6 is the only direct test: Random suffix yields OOD Avg 63.8, spurious suffix 64.2, a 0.4% difference on a single run with no error bars, while the jump from clean regularization to any suffix is 0.6%. With Lipsum-FT (random-text guidance) cited but never run, the evidence does not exclude the possibility that StarFT's gains come from generic KL-to-zero-shot regularization over augmented captions rather than from the semantic content of the spurious bank. The default bank is background-only (Sec. 4.5), yet the method improves style/texture shifts (ImageNet-Sketch, PACS) and data-source shift (CIFAR-10.02), which is more consistent with a content-agnostic regularizer than with coverage of task-specific spuriosity. If the 0.4% spurious-vs-random gap is noise, the paper's mechanistic claim fails, leaving a method that is close to Lipsum-FT.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StarFT, an auxiliary regularizer for contrastive fine-tuning of CLIP. Spurious descriptors are generated by querying a language model, appended to class captions, and the regularizer computes a KL divergence between the fine-tuned model and the frozen zero-shot model over spuriosity-augmented captions with true-class logits masked. Experiments on ImageNet OOD, group-shift, zero-shot transfer, and WILDS benchmarks report consistent improvements over FLYP, CAR-FT, and CaRot, with the largest gain being a 14.30% worst-group improvement on Waterbirds. Ablations attribute the gain to adding spurious suffixes, masking positive pairs, and decaying the regularization weight.","tokens_in":18932,"tokens_out":8351,"duration_ms":81441,"significance":"If the mechanism is as claimed, the paper would show that LLM-generated spurious text is a generic and cheap robustness regularizer, and it would provide a practical method that improves OOD, group-shift, zero-shot, and transfer performance simultaneously. The method is clearly specified, the code is released, and the evaluation spans many benchmarks. The authors also include a random-suffix ablation and a positive-pair-masking ablation, which are the right controls. However, the central mechanistic claim rests on a single ablation whose effect size is comparable to run-to-run noise, and the paper reports no error bars or multiple seeds; these gaps need to be closed before the significance can be assessed.","major_comments":[{"comment":"The only experiment that isolates the semantic content of the spurious descriptor bank is the comparison between the 'Random' and 'Spurious' suffix rows, which differ by 0.4% on ImageNet OOD average (63.8 vs 64.2) and by 0.4% on ImageNet (82.3 vs 82.7), on single runs with no error bars. Since the jump from clean-caption regularization to any suffix is 0.6% (63.2 to 63.8), the spurious-vs-random difference is too small to support the central claim that spuriosity alignment, rather than generic KL-to-zero-shot regularization over augmented captions, drives the reported gains. Please add multiple seeds with confidence intervals, and run Lipsum-FT (cited in Section 2 but never evaluated) as a direct baseline; if the gap is within noise, the mechanistic claim needs to be revised.","section":"Section 4.5, Table 6"},{"comment":"L_Star is a KL divergence between the fine-tuned model and the frozen zero-shot model over masked non-class logits, so the only place where spuriosity enters is the choice of the caption set S. The zero-shot model itself exhibits strong spurious bias (Table 3: Waterbirds zero-shot WG 25.9 vs Avg 87.1); aligning to its distribution is therefore not by itself a penalty on spurious features. To support the proposed mechanism, please provide a direct test that the regularizer specifically suppresses use of the spurious descriptors, e.g., an ablation with semantically plausible but non-spurious suffixes, or a quantitative measure of reliance on background versus foreground features.","section":"Section 3.2, Eq. (3)"},{"comment":"The default descriptor bank contains only the 'background' concept (Table 7), yet StarFT improves performance on style/texture shifts (ImageNet-Sketch, PACS) and data-source shift (CIFAR-10.02). The paper does not analyze whether the background bank covers the actual spurious cues in these benchmarks, so the observed benchmark pattern is at least as consistent with a content-agnostic regularizer as with a spuriousness-specific mechanism. Please add descriptor-coverage analysis or run a task-matched descriptor bank to show that matching the spurious cues matters.","section":"Section 4.5 and Appendix C.2"}],"minor_comments":[{"comment":"Equation (1) is referred to as '(3.1)' in several places, e.g., 'the contrastive loss (3.1)' and 'Eq. (3.1)'; please fix the equation references.","section":"Sections 3.1 and 3.2"},{"comment":"Please specify how the spurious descriptor is sampled for each caption (uniformly from the bank? once per image per batch?) and report the number of descriptors per concept; Appendix C.2 lists examples but not the sampling distribution.","section":"Section 3.2 and Appendix C"},{"comment":"The statement that the method is 'inherently robust' to LM hallucination is supported only by an intuition; either add a targeted experiment with intentionally misleading descriptors or soften the claim.","section":"Appendix D"},{"comment":"The lambda sweep reports ID/OOD values for three settings with no variance; since lambda=1.0 gives higher OOD than the default lambda=0.5, please state the explicit selection criterion for the default value.","section":"Appendix A.2, Table 10"},{"comment":"The sentence 'narrows the gap between the worst group and the average accuracy to 49.88%, which is 61.19% in zero-shot models' is confusing; the second number is the zero-shot gap (87.1 - 25.9), not a percentage of the model, and should be rephrased.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the journal and the method is cleanly presented. The main risk is that the claimed mechanism may not be distinguishable from Lipsum-FT-style random-text guidance; the authors should be asked to provide the Lipsum-FT comparison and multi-seed variance before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"StarFT is a clean, practical idea: during contrastive fine-tuning of CLIP, corrupt the label captions with LLM-generated spurious descriptors and align the fine-tuned model's masked-logit distribution to the frozen zero-shot model. The method consistently improves OOD, group-shift, zero-shot, and transfer metrics across many benchmarks, and the ablations are thoughtful. The code is released, and the masking of positive pairs is a genuinely nice trick that helps when the zero-shot teacher is weak (the iWildCam result is striking). I'd call it a useful addition to the robust fine-tuning toolbox, and the experiments are broad enough to take the headline results seriously.\n\nThe soft spots are real, though. The paper does not report error bars or multiple seeds anywhere, which matters a lot for worst-group accuracy: the Waterbirds jump from 21.5 to 40.2 is large, but a single run of a fine-tuned CLIP on a 2-class dataset could be noisy. More importantly, the closest baseline, Lipsum-FT, is cited but never run, and the direct test of the mechanism—Table 6—shows only a 0.4% OOD-average difference between random suffixes and spurious suffixes. With no variance estimate, that gap could easily be noise. If it is noise, the paper's central claim (that the semantic content of spurious descriptors is what prevents shortcut learning) collapses, and StarFT becomes something close to Lipsum-FT with a positive-pair mask. That is still a decent contribution, but the title and framing would be overstated.\n\nI also think the descriptor bank's role is under-analyzed. The default is background-only, yet the method improves ImageNet-Sketch and PACS, which are style/texture shifts. That pattern is more consistent with generic random-text regularization than with targeted spurious-feature suppression. The paper doesn't validate coverage or overlap of the spurious bank with actually diagnostic features. These are addressable concerns, not fatal ones.\n\nOverall: the method works in practice, the experiments are honest (they show trade-offs, e.g., slight ID loss with stronger lambda), and the limitations section is candid. The main weaknesses are missing variance information and an absent head-to-head with Lipsum-FT. I would encourage the authors to add seeded runs and that baseline before publication.\n\nThis paper deserves serious peer review. It is practically relevant, clearly written, and reproducible enough that a good referee can push on the right questions. I'd bring it to reading group and would cite it if I worked in this area. My recommendation: send it to review, with the expectation that the mechanistic claim needs either stronger evidence or a softer interpretation.","headline":"A practically effective robust fine-tuning regularizer whose headline mechanism—spurious semantics driving the gains—is weakly supported, since the random-suffix gap is tiny and no error bars or Lipsum-FT comparison are provided.","tokens_in":19492,"tokens_out":2724,"would_cite":true,"duration_ms":27335,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StarFT claims that aligning fine-tuned CLIP's masked-logit distribution over LM-generated spurious captions to the frozen zero-shot model prevents spurious feature learning and preserves robustness.","keywords":["robust fine-tuning","zero-shot models","CLIP","spurious correlation","shortcut learning","textual alignment regularization","group robustness","distribution shift"],"falsifier":"Take a dataset whose known spurious cue is deliberately absent from the LM descriptor bank, for example Waterbirds with only texture and resolution descriptors and no background descriptors, and fine-tune with StarFT; if worst-group accuracy on the water/land subgroups still rises substantially, the background descriptors are not the active mechanism, and if it does not rise, the mechanism is descriptor coverage rather than generic regularization.","tokens_in":18419,"feed_emoji":"🐦","tokens_out":5605,"duration_ms":49868,"temperature":0.7,"pith_summary":"StarFT claims that the drop in out-of-distribution robustness seen when fine-tuning zero-shot models such as CLIP can be traced to spurious feature learning, and that this can be countered by a single textual regularizer. The regularizer builds captions that append language-model-generated spurious descriptors (for example, \"in the mountains\" or \"with a rough texture\") to the class prompt, then forces the fine-tuned model's masked-logit distribution over those captions to match the frozen zero-shot model's distribution. If the claim is right, robustness is not a property that must be preserved wholesale; it can be protected by telling the model which confounds not to rely on. On Waterbirds, StarFT improves worst-group accuracy by 14.30 percentage points and average accuracy by 3.02 points over the FLYP baseline, where competing robust fine-tuning methods degrade.","feed_headline":"Regularizer stops fine-tuned CLIP from learning background shortcuts","feed_subtitle":"StarFT's masked-KL loss against LM-generated spurious captions adds 14.3 points on Waterbirds' hardest group.","key_machinery":"The load-bearing object is the spurious textual alignment loss $\\mathcal{L}_{\\mathrm{Star}}$, computed on a batch of spuriosity-augmented captions $S_i$ formed by appending LM-generated descriptors to each class template. For each image, the logits of both the fine-tuned and frozen zero-shot models are masked to remove the true-class column, so the softmax $q_i$ (fine-tuned) and $\\tilde{q}_i$ (zero-shot) encode only relative likelihoods among spurious descriptions; the loss is $\\frac{1}{N}\\sum_i D_{\\mathrm{KL}}(\\tilde{q}_i \\| q_i)$. This masked-KL form is what prevents the fine-tuned model from learning to lean on the spurious descriptions, while the true-class masking keeps an imperfect zero-shot teacher from injecting erroneous confidence.","core_discovery":"The paper's central claim is that minimizing the spurious textual alignment loss $\\mathcal{L}_{\\mathrm{Star}}$ (Eq. 3), a KL divergence between the fine-tuned and frozen zero-shot softmax distributions over spuriosity-augmented captions with true-class logits masked, prevents the fine-tuned model from further extracting the spurious features named by those captions. Masking the true class avoids distilling the zero-shot model's confident but wrong positive-pair predictions when it is weak; decaying the regularization weight balances in-distribution accuracy against out-of-distribution robustness. The authors argue that this mechanism, rather than prompt or context preservation, explains why StarFT improves OOD averages on ImageNet variants, worst-group accuracy on Waterbirds, PACS, and CIFAR-10.02, zero-shot accuracy on four natural-image benchmarks, and transfer average rank, all without extra data or task-specific prompts.","pith_inferences":["If descriptor coverage drives the gains, an oracle bank that names the exact dataset confound should make StarFT nearly solve the group-shift benchmark, while a bank that omits it should erase the improvement; this comparison would isolate the mechanism from generic regularization.","The masked-logit distribution could double as an audit tool: tracking per-descriptor KL movement during fine-tuning would reveal which confounds the model is actually learning to suppress.","For datasets whose confounds are not generic natural-image cues (medical imaging, satellite imagery), the method would need automatic per-task spuriosity discovery; the paper leaves that extension open."],"forward_implications":["If the central claim holds, robust fine-tuning can be achieved by specifying what not to learn, rather than only by preserving the pretrained model's weights or contexts.","A single generic descriptor bank transfers across benchmarks: fine-tuning on ImageNet with StarFT gives zero-shot worst-group gains on Waterbirds, PACS, and CIFAR-10.02 without those datasets appearing in training.","StarFT improves worst-group and average accuracy together, which goes against the usual expectation that removing spurious reliance must sacrifice average performance.","Because the frozen zero-shot model acts as teacher, StarFT does not inherit the EMA-based failure mode of CaRot on datasets where the zero-shot model is weak, such as WILDS-iWILDCam."],"supporting_citations":[{"why":"Supplies the CLIP zero-shot model and prompting setup that StarFT fine-tunes and distills from.","marker":"[Radford et al., 2021]"},{"why":"Provides the FLYP contrastive fine-tuning objective that StarFT builds on as its base loss.","marker":"[Goyal et al., 2023]"},{"why":"Defines the robust fine-tuning setting and the training protocols and weight-ensembling baseline used for comparison.","marker":"[Wortsman et al., 2022]"},{"why":"Supplies the Waterbirds dataset and worst-group accuracy metric central to the group-shift experiments.","marker":"[Sagawa et al., 2020]"},{"why":"Motivates the shortcut-learning account that spurious features cause robustness loss during fine-tuning.","marker":"[Geirhos et al., 2020]"},{"why":"Defines the CaRot baseline that StarFT compares against, especially its EMA-based self-distillation behavior.","marker":"[Oh et al., 2024]"},{"why":"Provides the keyword-based method used to identify ImageNet spurious correlations such as snow for husky and sand for rattlesnake.","marker":"Kim et al. [2024]"},{"why":"Supplies the language model used to generate the spurious concept descriptors and spuriosity-augmented captions.","marker":"[OpenAI, 2023]"}],"fun_headline_variants":["Masked KL with LLM spurious captions lifts CLIP worst-group 14.3%","StarFT regularizes CLIP against spurious text, adding 14.3% on hardest group","Fine-tune CLIP without spurious features: 14.3% worst-group boost","LLM-generated spurious captions, masked KL: StarFT beats robust baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method only suppresses the spurious cues written in the language-model-generated descriptor bank, so the bank must actually cover the confounds a dataset exploits; if it misses the real bias or overlaps with features that genuinely define the class, the promised robustness gain will not appear or in-distribution learning will suffer.","fun_headline_variants_meta":{"raw":{"variants":["Masked KL with LLM spurious captions lifts CLIP worst-group 14.3%","StarFT regularizes CLIP against spurious text, adding 14.3% on hardest group","Fine-tune CLIP without spurious features: 14.3% worst-group boost","LLM-generated spurious captions, masked KL: StarFT beats robust baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3976,"prompt_tokens":987,"completion_tokens":2989,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2891}},"tokens_in":603,"tokens_out":2989,"duration_ms":21742,"temperature":1.0,"reasoning_tokens":2891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:44.365110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset whose known spurious cue is deliberately absent from the LM descriptor bank, for example Waterbirds with only texture and resolution descriptors and no background descriptors, and fine-tune with StarFT; if worst-group accuracy on the water/land subgroups still rises substantially, the background descriptors are not the active mechanism, and if it does not rise, the mechanism is descriptor coverage rather than generic regularization.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP zero-shot model and prompting setup that StarFT fine-tunes and distills from."},{"cited_title":"Finetune like you pretrain: Improved finetuning of zero-shot vision models","cited_arxiv_id":null,"evidence_quote":"Provides the FLYP contrastive fine-tuning objective that StarFT builds on as its base loss."},{"cited_title":"Robust fine-tuning of zero-shot models","cited_arxiv_id":null,"evidence_quote":"Defines the robust fine-tuning setting and the training protocols and weight-ensembling baseline used for comparison."},{"cited_title":"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the Waterbirds dataset and worst-group accuracy metric central to the group-shift experiments."},{"cited_title":"Shortcut learning in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Motivates the shortcut-learning account that spurious features cause robustness loss during fine-tuning."}],"review_version":1}