{"id":"a0e915a8-0c3c-442b-9dc1-d4e37b70666f","arxiv_id":"2411.16236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DoubleCCA merges CLIP text prompts with sentence-embedding-model features of random prompt variants via two CCA steps, improving worst-group accuracy of zero-shot classifiers without group labels or image data.","lead":"This paper combines random text prompts with a second sentence-embedding model and two rounds of Canonical Correlation Analysis to build merged class prompts for zero-shot CLIP classification. The method raises worst-group accuracy on Waterbirds and CelebA in several settings, but the gains are uneven and come with caveats about error bars, baseline comparisons, and a proxy distribution assumption.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The merger in Eq. 5 is fit on random-sentence text embeddings assumed to simulate images (Sect. 3.2.2); this proxy is unvalidated and the stated dimensions are internally inconsistent, so the headline robustness gains are not yet supported.","rationale":"The reader identified the proxy-distribution assumption in Section 3.2.2 as the weakest link, and I agree that it is the conceptual core of the paper: the second CCA is calibrated on random-sentence CLIP embeddings and then applied to real image embeddings at test time. That is exactly the load-bearing premise. I add one concrete internal issue that makes the concern more severe: as written, the matrix dimensions in Eq. 6 do not close under the stated hyperparameter setting in Section 4.1. For Waterbirds with two classes and K=500, the second CCA cannot have dimension 1024 under either natural orientation of Fr, so Algorithm 1 is not fully specified. This does not prove the method is wrong, but it means the empirical numbers cannot be reproduced or checked from the paper alone. I therefore keep the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT: the idea is coherent, builds on plausible ingredients, and the paper honestly reports trade-offs, but the central proxy step needs a corrected derivation, code, and a direct validation against real image embeddings before the robustness claim can be accepted.","tokens_in":12149,"tokens_out":14632,"duration_ms":288172,"concrete_test":"Re-implement Algorithm 1 exactly as written for Waterbirds RN50 (C=2, K=500, dCCA=1024). If the CCA fit fails, warns about rank, or produces a degenerate M, the equations as stated are inconsistent. Then, with whatever orientation the authors intend, fit the second CCA on real Waterbirds image embeddings instead of Fr and compare the worst-group accuracy of the resulting W against the reported one; a material change (more than about 5 points) would falsify the proxy assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 and Algorithm 1 step 7 use XA = Ŵx Fr and XB = Ŵse Fr, with Fr described as simulating image representations, to fit the second CCA and hence M in Eq. 5. At inference, W is applied to real image embeddings. This is the only bridge from text-only calibration to image classification, and it is asserted, not tested. The problem is not only empirical: with Ŵx, Ŵse ∈ R^{n_classes×d} and Fr ∈ R^{n_random×d} or its transpose, the stated choice \"the dimension of the second CCA is set to the dimension of the original image embeddings\" (Section 4.1) is dimensionally impossible for Waterbirds RN50 (n_classes=2, d=1024, K=500 so n_random=1000). As written, XA and XB are either oriented so CCA sees only 2 samples, or the requested CCA dimension exceeds the rank of both matrices. Thus the central construction is underspecified, and the reported worst-group improvements in Table 1 cannot be attributed to the described mechanism without code or a corrected derivation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DoubleCCA, a training-free method to improve the worst-group robustness of CLIP zero-shot classifiers. The method generates K random sentences per class, embeds them with both the CLIP text encoder and an external sentence embedding model (HiT), and then applies CCA twice: first to align the two text embedding spaces, and second to merge the two resulting score functions into a single text embedding matrix. At inference, this merged matrix is used to score CLIP image embeddings. Experiments on Waterbirds and CelebA report improved worst-group accuracy across four backbones, and results on six domain-generalization datasets show maintained or slightly improved accuracy. The central claim is that random-sentence text enrichment plus CCA merging improves group robustness without using image data, group labels, or training.","tokens_in":12274,"tokens_out":5376,"duration_ms":46812,"significance":"If the central claim holds, the method is an interesting training-free debiasing recipe for CLIP: it requires no image data or group labels, is conceptually simple, and can be combined with existing adapters. The paper includes useful ablations on the sentence embedding model and on hyperparameters, and it addresses a relevant problem. However, the evidence is currently weak: no code, no seeds, and no error bars are reported; the Waterbirds worst-group test set is very small; and there is a load-bearing dimension inconsistency in the second CCA construction. The assumption that random-sentence text features can simulate image representations is asserted but not validated, leaving the mechanism unsupported. These issues must be resolved before the reported robustness gains can be taken as established.","major_comments":[{"comment":"The construction XA = Ŵx Fr, XB = Ŵse Fr is dimensionally inconsistent with the stated experimental settings. For Waterbirds with RN50, n_classes=2, d=1024, and K=500, Fr has 1000 rows. If Fr is n_random×d, the products ŴxFr and ŴseFr are undefined because Ŵx and Ŵse are 2×1024; if Fr is d×n_random, then XA and XB are 2×1000, so after centering the second CCA has rank at most 1 and cannot produce a common space of dimension equal to the original image embedding dimension (1024), as stated in Section 4.1. The paper must specify the exact shapes of Fr, XA, XB, the centering and regularization used in the second CCA, and ideally release code; otherwise the reported numbers cannot be reproduced or attributed to the described mechanism.","section":"Section 3.2.2 / Algorithm 1 step 7"},{"comment":"The statement that random sentence embedding features Fr 'simulate the image representations' is the sole bridge between text-only calibration and image inference, but it is asserted without evidence. The second CCA is fit on XA = Ŵx Fr and XB = Ŵse Fr, i.e., on the text-based score functions evaluated on random sentences, while at test time the merged matrix W is applied to real image embeddings. This is a domain shift that the paper does not validate. To support the central claim, the authors should provide quantitative evidence that Fr-derived features are representative of image features in the relevant space, for example by comparing distributions, or by ablating the second CCA with real image features from a validation set. Without such validation, the reported worst-group improvements cannot be confidently assigned to the proposed calibration mechanism.","section":"Section 3.2.2"},{"comment":"No seeds, error bars, or confidence intervals are reported anywhere. The Waterbirds worst-group test split is small (about 56 images), so the headline RN50 improvement from 16.07% to 44.64% corresponds to a change of roughly 9 to 25 correct images, which could arise from sampling noise. Please report means and standard deviations over at least 5 random seeds for all reported numbers, and state the exact sizes of the worst-group test sets for Waterbirds and CelebA. The same applies to the domain generalization results in Table 2.","section":"Section 4.1 / Table 1"},{"comment":"The text compares DoubleCCA with PerceptionCLIP and FairerCLIP, but these baselines do not appear in Table 1, and the reported metric 'gap' is never defined (presumably average accuracy minus worst-group accuracy). Please include the baselines in the table or explicitly state that the comparison is qualitative, and define 'gap' when it is first used. Without the baselines, the claims of outperforming existing methods are not directly supported by the presented experiments.","section":"Section 4.2 / Table 1"}],"minor_comments":[{"comment":"The symbol Fr is reused in two different roles: first as the random-sentence embedding matrix for the first CCA, and then as the proxy for image representations in the second CCA. This conflates two distinct objects and contributes to the dimension ambiguity; please use different names, e.g., F_r^{text} for the first CCA and F_r^{proxy} for the second.","section":"Section 3.2.1 and 3.2.2"},{"comment":"The phrase 'use CCA double twice' should be corrected to 'use CCA twice'.","section":"Abstract"},{"comment":"There are several typos: 'backgroup' should be 'background', 'folowers' should be 'flowers', and 'with folowers' appears in the list of background contexts. The heading in Section 4.5, 'Abalation Study', should be 'Ablation Study'.","section":"Section 4.2"},{"comment":"In the caption of Figure 6(b), 'Dimension of CCA' is ambiguous: Section 4.1 sets the first CCA dimension to 64, while the ablation varies the dimension up to 384. Please specify which CCA (first or second) is being varied, and clarify whether the second CCA dimension is also fixed or follows the stated rule.","section":"Figure 6"},{"comment":"The paper says it follows 'the same experimental settings as the previous work [11]' but does not specify which elements are taken from that work (prompt templates, data splits, evaluation protocol). Please list the exact settings used, including the prompt template and the source of the random sentences, to make the experiments reproducible.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is not reproducible as written: there is no code, no seeds, and the dimension inconsistency in Section 3.2.2 prevents reimplementation. I strongly recommend that the editor ask for a corrected derivation, exact tensor shapes, and code before any acceptance decision. The novelty relative to WaffleCLIP and PerceptionCLIP should also be clarified with direct comparisons rather than text-only claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper combines two known ingredients—WaffleCLIP's random-character prompt augmentation and CCA-based merging of classifiers from Horoi et al.—into a training-free debiasing recipe for zero-shot CLIP classifiers. That specific combination is new, and the setup is appealing: no image data, no group labels, no adapter training. If the headline Waterbirds worst-group gains (16→45 on RN50) hold, it's a genuinely convenient knob for practitioners.\n\nWhat the paper does well: the ablations are thoughtful. It tests several sentence-embedding models, the number of random sentences, and the CCA dimension. It also honestly reports that the first CCA alone helps and that the second CCA adds further gains. The related-work placement is fair, crediting WaffleCLIP and Horoi et al. clearly.\n\nThe soft spots are in the evidence. No code, no seeds, no error bars anywhere. The Waterbirds worst-group test set has only ~56 images, so the headline deltas are literally a handful of correct predictions. The hyperparameters (K=500, dim=64) are tuned on Waterbirds—the same benchmark that carries the main result. The abstract claims \"outperforms existing methods,\" but Tables 1 and 2 only compare to vanilla CLIP; WaffleCLIP itself isn't in the main table. And the paper's own tables show the average accuracy often drops slightly, which the text acknowledges as a trade-off but the abstract ignores.\n\nThe bigger structural concern is the second CCA's calibration data. Section 3.2.2 and Algorithm 1 use random-sentence text embeddings as stand-ins for image representations when fitting the merger M. That proxy assumption is asserted, not validated. Worse, the dimensions don't obviously work: for Waterbirds RN50, n_classes=2, K=500 so n_random=1000, and d=1024. Setting the second CCA dimension to d=1024 is impossible—CCA cannot return more components than the rank of the centered data matrices, which is at most 1000 (or 1, if the matrices are oriented the way the equations suggest). This could be a typo or an implementation detail, but as written it means the described mechanism can't reproduce the reported numbers. That needs to be fixed before the result is credible.\n\nVerdict: this is a promising idea that deserves to be iterated on, but the paper as posted is not ready. A serious referee should see it—the combination is meaningful and the flaws are addressable—but they should ask for code, seeds, variance over random-sentence draws, a direct comparison with WaffleCLIP, and a corrected derivation of the second CCA. I'd bring it to a reading group to discuss the proxy assumption, but I wouldn't cite it yet.","headline":"A clever, training-free debiasing recipe combining WaffleCLIP prompts with CCA merging, but the empirical support is under-powered and the second CCA's geometry is underspecified.","tokens_in":12938,"tokens_out":2821,"would_cite":false,"duration_ms":25245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a text-only, training-free CCA-based merge of CLIP and sentence-embedding representations improves worst-group zero-shot accuracy, with Waterbirds ResNet-50 worst-group accuracy rising from 16.07% to 44.64%.","keywords":["group robustness","CLIP","zero-shot classification","canonical correlation analysis","sentence embeddings","random prompt augmentation","spurious correlations","text-only debiasing"],"falsifier":"Run DoubleCCA with the second alignment fitted on real image embeddings from a held-out split (or on dimension-matched random noise) instead of random-sentence embeddings and compare worst-group accuracy: if the gains disappear or change materially, the random-sentence proxy is doing the work; if not, the claimed alignment mechanism is not the cause.","tokens_in":11784,"feed_emoji":"⚖️","tokens_out":8862,"duration_ms":73965,"temperature":0.7,"pith_summary":"The paper proposes DoubleCCA, a training-free recipe for making CLIP zero-shot classifiers more robust to group-based biases, such as a background that correlates with the label. The idea is to augment each class prompt with many random sentences, embed them with both the CLIP text encoder and a separate sentence embedding model, and then align and merge the two embedding spaces with two passes of canonical correlation analysis (CCA). The result is a single merged text-embedding matrix that replaces the original prompt embeddings at inference time. The paper reports that on Waterbirds with a ResNet-50 backbone, worst-group accuracy rises from 16.07% to 44.64% with ordinary prompts and from 39.29% to 48.21% when background context is included, with smaller gains on CelebA and with domain-generalization accuracy roughly maintained across six datasets. If correct, this would mean group robustness can be improved with text alone, without image data, group labels, or fine-tuning.","feed_headline":"Double CCA more than doubles CLIP's worst-group accuracy on Waterbirds","feed_subtitle":"A training-free, text-only recipe that improves worst-group accuracy while keeping domain generalization intact.","key_machinery":"The central object is the merged text-embedding matrix $W = \\tfrac{1}{2}(\\hat{W}_x + M\\hat{W}_{se})$, where $\\hat{W}_x$ and $\\hat{W}_{se}$ are the reconstructed score-function weights from the first CCA and $M = (P_B P_A^{-1})^T$ is the merger learned by the second CCA. Canonical correlation analysis (CCA) is the statistical method that finds linear projections maximizing correlation between two feature sets; here it is applied twice. The first pass aligns CLIP text embeddings with sentence-embedding-model outputs on a corpus of random sentences, and the second pass merges the two score functions using random-sentence features as stand-ins for image embeddings, since no images are available. The machinery carries the argument by turning an uncalibrated random-prompt augmentation into a calibrated linear merge that plugs directly into the existing CLIP zero-shot score $\\hat{y} = \\arg\\max_y W_y \\Phi_v(I)$.","core_discovery":"The central claim is that canonical correlation analysis can turn noisy random-prompt text embeddings into a more group-robust zero-shot classifier by merging two text-embedding sources. Specifically, the paper constructs the merged text embedding matrix $$W = \\tfrac{1}{2}(\\hat{W}_x + M\\hat{W}_{se}), \\qquad M = (P_B $P_A^{{-1}}$)^T,$$ where the first CCA aligns the CLIP text encoder with a sentence embedding model on random-sentence features, and the second CCA merges the two resulting score functions into one. Inference is then unchanged in form: $\\hat{y} = \\arg\\max_y W_y \\Phi_v(I)$. The paper argues that this merged embedding improves worst-group accuracy because the extra sentence encoder supplies semantically meaningful structure that the CLIP text encoder lacks, while the CCA alignment prevents the random tokens from collapsing class separability, as WaffleCLIP does on smaller backbones. The empirical case is made on Waterbirds and CelebA across four CLIP backbones, plus six domain-generalization datasets.","pith_inferences":["A direct test of the paper's mechanism: fit the second CCA on a held-out set of real image embeddings instead of random-sentence proxies. If the worst-group gains vanish, the proxy assumption is doing the work; if they persist, the paper's stated role of random sentences as image stand-ins is not the active ingredient.","Because the merged embedding is computed entirely from text, the recipe suggests a path to debiasing open-vocabulary or newly added classes on the fly, as long as a sentence encoder is available; this extension is not tested in the paper.","The ablation curve with sentence count implies the random-sentence distribution is the carrier of robustness. One could therefore try structured or semantic perturbations of the random sentences (e.g., class-related nouns instead of characters) and expect larger worst-group gains, though the paper does not explore this."],"forward_implications":["If the central claim holds, zero-shot CLIP classification can be made more group-robust without any image data, group annotations, or gradient updates, simply by substituting $W$ for the original text embeddings.","Because $W$ is built only from class names and random sentences, the recipe transfers across backbones (RN50, ViT-B/32, ViT-B/16, ViT-L/14) and datasets, improving worst-group accuracy on Waterbirds and CelebA while keeping domain-generalization accuracy roughly constant.","The merged embeddings can be dropped into existing adapter-based debiasing pipelines: combining DoubleCCA with a contrastive adapter further lifts worst-group accuracy, so the text-only enrichment is complementary rather than exclusive.","Sufficient random sentences (about 500 per class) are required for stable worst-group gains; with fewer random sentences the method can fall below the CLIP baseline, implying that the text-only augmentation distribution, not the class descriptions alone, carries the robustness signal."],"supporting_citations":[{"why":"Defines the CLIP vision-language model whose text and image encoders and zero-shot similarity scoring are the object being improved.","marker":"[1]"},{"why":"Introduces LLM-generated class descriptions, the prior approach whose ambiguity motivates replacing descriptions with random sentences.","marker":"[9]"},{"why":"WaffleCLIP, the source of random word/character prompt augmentation and the baseline whose instability DoubleCCA is designed to fix.","marker":"[10]"},{"why":"PerceptionCLIP, which supplies the contextual-attribute prompting setup and the baseline for the background/gender augmented experiments.","marker":"[11]"},{"why":"Contrastive adapters, the group-robustness baseline and the method DoubleCCA is combined with to show complementarity.","marker":"[12]"},{"why":"The Hierarchy Transformer sentence embedding model used to produce the complementary text embeddings merged with CLIP.","marker":"[21]"},{"why":"Supplies the regularized CCA formulation used to make the transformation computations more stable.","marker":"[22]"},{"why":"The CCA-based neural network merging procedure that motivates the second CCA and the merger matrix M.","marker":"[23]"},{"why":"The Waterbirds dataset, the primary group-robustness benchmark used for the main comparisons and ablations.","marker":"[24]"},{"why":"The CelebA dataset, the second group-robustness benchmark used for the main results.","marker":"[27]"}],"fun_headline_variants":["DoubleCCA uses CCA on random sentences to double CLIP worst-group accuracy","Random sentences plus CCA boost CLIP robustness on Waterbirds and CelebA","Training-free text tweak: DoubleCCA improves group fairness without retraining","Merging CLIP and sentence embeddings via CCA lifts worst-group accuracy","DoubleCCA: CCA on random prompts doubles worst-group accuracy in CLIP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recipe's gains rest on the assumption that text embeddings of random sentences behave like real image embeddings during the second alignment step, since the method never looks at an actual image.","fun_headline_variants_meta":{"raw":{"variants":["DoubleCCA uses CCA on random sentences to double CLIP worst-group accuracy","Random sentences plus CCA boost CLIP robustness on Waterbirds and CelebA","Training-free text tweak: DoubleCCA improves group fairness without retraining","Merging CLIP and sentence embeddings via CCA lifts worst-group accuracy","DoubleCCA: CCA on random prompts doubles worst-group accuracy in CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1405,"prompt_tokens":915,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":531,"tokens_out":490,"duration_ms":5299,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:22:35.292181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DoubleCCA with the second alignment fitted on real image embeddings from a held-out split (or on dimension-matched random noise) instead of random-sentence embeddings and compare worst-group accuracy: if the gains disappear or change materially, the random-sentence proxy is doing the work; if not, the claimed alignment mechanism is not the cause.","supporting_citations":[{"cited_title":"Waffling around for performance: Visual classification with random words and broad concepts","cited_arxiv_id":null,"evidence_quote":"WaffleCLIP, the source of random word/character prompt augmentation and the baseline whose instability DoubleCCA is designed to fix."},{"cited_title":"PerceptionCLIP: Visual classification by inferring and conditioning on contexts","cited_arxiv_id":null,"evidence_quote":"PerceptionCLIP, which supplies the contextual-attribute prompting setup and the baseline for the background/gender augmented experiments."},{"cited_title":"Contrastive adapters for foundation model group robustness","cited_arxiv_id":null,"evidence_quote":"Contrastive adapters, the group-robustness baseline and the method DoubleCCA is combined with to show complementarity."},{"cited_title":"Language models as hierarchy encoders","cited_arxiv_id":null,"evidence_quote":"The Hierarchy Transformer sentence embedding model used to produce the complementary text embeddings merged with CLIP."},{"cited_title":"Eigenproblems in pattern recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the regularized CCA formulation used to make the transformation computations more stable."},{"cited_title":"Harmony in diversity: Merging neural networks with canonical correlation analysis","cited_arxiv_id":null,"evidence_quote":"The CCA-based neural network merging procedure that motivates the second CCA and the merger matrix M."}],"review_version":1}