{"id":"4cfd0c43-6232-4ba5-9e10-7034d728821e","arxiv_id":"2509.06535","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FairCLIP's claimed fairness and performance gains over CLIP do not reproduce on two datasets, and its official implementation diverges from the paper's own formulation.","lead":"This paper reruns FairCLIP, a published method meant to make the CLIP image-text model fairer in medical imaging, and finds that the official code does not match the method's own description and selects its best model on the test set. In the authors' hands, neither the original code nor a corrected reimplementation improves fairness or accuracy in zero-shot glaucoma classification, even though the regularizer does shrink its target distances.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Three-seed runs make the central negative claim fragile: most FairCLIP-vs-CLIP-FT differences are smaller than their standard deviations, so 'no improvement found' may be a power artifact rather than evidence of absence.","rationale":"The reader's weakest_assumption centers on protocol fidelity: validation-based selection, aligned similarity computation, 500-epoch linear probes, and hyperparameters. Those are real limitations, but the single most load-bearing vulnerability is statistical. Even if every protocol deviation were corrected, three-seed means with standard deviations exceeding the reported differences cannot sustain a negative universal claim. The paper itself acknowledges this in Section 5.1, where it says standard deviations may be stabilized by increasing the number of runs. This is load-bearing because the central claim is negative; absence of evidence requires demonstrated sensitivity. I would not reject the paper: it is carefully hedged, ships code, and documents the official implementation's test-set selection. The negative conclusion should be read as provisional, consistent with a conditional verdict. My agreement with the reader is partial because their rationale names underpowering but their weakest_assumption field emphasizes protocol fidelity, which I see as secondary to the power problem.","tokens_in":26675,"tokens_out":8040,"duration_ms":95821,"concrete_test":"Run the ViT-B/16 A-FairCLIPgender versus CLIP-FT comparison on Harvard-FairVLMed with at least 10 additional seeds (same official data splits and validation-set selection), compute per-seed DPD, DEOdds, AUC, and ES-AUC differences, and report 95% bootstrap confidence intervals and a paired test. If the CI for the DPD/ES-AUC difference excludes zero in FairCLIP's favor, the negative claim fails; if it includes zero or favors CLIP-FT, the claim stands with the required precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that FairCLIP does not improve performance or fairness—is an absence claim, and the experiments are too underpowered to support it. Every condition uses only three seeds (Section 3.3), and the paper itself notes in Section 5.1 that standard deviations are relatively high and could be stabilized by increasing the number of runs. In Table 2, key comparisons are often within one standard deviation: A-FairCLIPgender DPD is 1.57±2.13 vs CLIP-FT 2.97±3.08, and AUC is 70.00±1.94 vs 70.14±1.63; A-FairCLIPlanguage AUC is 71.04±1.99 vs 70.14±1.63 with DPD worse. On FairFace (Table 6), differences are similarly on the order of 0.1–1 AUC points with standard deviations of 0.2–5. Without a power analysis or confidence intervals on the differences, the observed non-improvement cannot be distinguished from a failure to detect real gains. This is especially acute because the original reported effects were large (Appendix D, Table 10: FairCLIPgender DPD 0.84±0.25 vs CLIP-FT 4.34±0.66), while the reproduction reverses the ordering (5.69±1.23 vs 2.97±3.08). With n=3 and no multiple-comparison control, such a reversal is not decisive. The conclusion is consistent with the data but not robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a reproducibility study of FairCLIP (Luo et al., 2024), a method that fine-tunes CLIP with a Sinkhorn-distance regularizer to improve group fairness in zero-shot glaucoma classification. The authors document that the official FairCLIP implementation deviates from the model description in three ways: group similarity scores are normalized by their sum, the similarity matrix uses IT⊤T I⊤ rather than diag(IT⊤), and model selection is performed on the test set rather than the validation set. They introduce an aligned implementation (A-FairCLIP) and a multi-attribute extension (FairCLIP+), and evaluate on Harvard-FairVLMed (ViT-B/16 and ViT-L/14) and FairFace (RS50). Across these settings, they find that FairCLIP, A-FairCLIP, and FairCLIP+ generally do not improve AUC or fairness metrics over fine-tuned CLIP, even though the regularization objective reduces Sinkhorn distances. The paper concludes that Claim 2 of the original paper—that FairCLIP improves both performance and fairness—is not supported by their experiments, while Claim 1 (CLIP exhibits demographic biases) receives weak support.","tokens_in":26918,"tokens_out":5200,"duration_ms":56165,"significance":"If the negative result is robust, this would be an important caution for practitioners and would cast doubt on the original CVPR 2024 results. The paper's concrete documentation of the implementation–description mismatch is a genuine contribution, as is the distance-reduction-without-fairness finding. The authors also provide code and detailed computational cost accounting, which strengthens the reproducibility value. However, the central claim is an absence claim and is supported only by three seeds per condition with large standard deviations; many key comparisons are within one standard deviation. As reported, the evidence is too underpowered to definitively overturn the original claims, though it is a legitimate and useful negative result.","major_comments":[{"comment":"The central negative claim is underpowered. All experiments use three seeds (Section 3.3), and Section 5.1 itself acknowledges that standard deviations are relatively high and could be stabilized by more runs. In Table 2, A-FairCLIPgender DPD is 1.57±2.13 vs CLIP-FT 2.97±3.08, and AUC is 70.00±1.94 vs 70.14±1.63; these differences are far smaller than the noise. Table 6 shows AUC differences of 0.1–1 point with standard deviations of 0.2–5. No confidence intervals, effect sizes, or power analysis is provided. Since “do not support Claim 2” is an absence claim, the current design cannot distinguish a true absence of improvement from a failure to detect a real gain.","section":"3.3, 4.1.2, 5.1"},{"comment":"Several protocol deviations make the comparison not fully apples-to-apples with the original study. Linear probes are trained for 500 instead of 1000 epochs; FairFace fine-tuning uses a random 11,000-sample subset, custom prompts, and the small RS50 architecture; and A-FairCLIP hyperparameters are tuned on the gender attribute only and then applied to all attributes. The inversion in Table 10 (FairCLIPgender DPD 5.69±1.23 vs the original 0.84±0.25) could be due to these choices rather than to the method itself. The authors should justify each deviation or provide a sensitivity analysis showing it does not drive the negative conclusion.","section":"3.4, Appendix B"},{"comment":"Because the original implementation selected the best model on the test set, the paper’s corrected validation-based protocol is a modified version of FairCLIP. The conclusion that the experimental results do not support Claim 2 is therefore about the corrected protocol, not about the original published pipeline as actually implemented. The paper notes this in Section 5.1, but the abstract and Section 5.1 should be more explicit that the official-code reproduction also changed model selection; otherwise readers may over-interpret the negative result as directly refuting the original paper.","section":"3.3, 5.1"},{"comment":"The claim that minimizing Sinkhorn distance does not imply improved fairness rests on a small number of runs and a single distance metric. Tables 4 and 5 show large reductions in distance, but the accompanying fairness metrics have high variability. A formal test of association between distance reduction and fairness improvement, or at least confidence intervals on the distances, would be needed to support the conclusion that the regularizer’s effect is purely on the Sinkhorn distance without downstream benefit.","section":"4.2.2, Tables 4-5"}],"minor_comments":[{"comment":"Equation (2) writes d(DB,DB) but from the surrounding text and Equation (1) it should presumably be d(DB,DBa). Please correct the typo.","section":"3.1, Eq. (2)"},{"comment":"In the Male row of the Gaussian MMD table (Table 13/14), the entry “3.55·104” appears to be a typo for “3.55·10−4”; as printed it is inconsistent with the neighboring values.","section":"Appendix E, Table 14"},{"comment":"The sentence “since there were no significant differences between those parameters and the parameters found with gender” is self-referential; presumably one of the two mentions of ‘gender’ should be ‘ethnicity’ or another attribute.","section":"Appendix B"},{"comment":"Minor typo: “constrastive” should be “contrastive”.","section":"3.4.2"},{"comment":"Grammar: “The evaluations was done” should be “The evaluations were done”.","section":"3.4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped reproducibility report with a concrete and valuable implementation-mismatch analysis. The main weakness is statistical power: the absence claim is supported by n=3 seeds and many high-variance comparisons. The paper is potentially publishable after adding interval estimates or a power analysis, and after clarifying the protocol-deviation caveats. I would not reject on the basis that the findings disagree with the original paper; the disagreement is itself informative, but it needs to be presented with appropriate statistical humility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid, transparent reproducibility study with one concrete finding that matters more than the negative result itself: the official FairCLIP implementation does not match the published formulation. The group-score normalization, the diag(I T^T T I^T) similarity instead of diag(I T^T), and especially the test-set model selection are all documented precisely and are easy to verify. That last one is a serious problem for the original paper, and this study is right to flag it.\n\nWhat the paper does well: it ships code, discloses every protocol deviation, runs two implementations (official and aligned), two architectures, and two datasets, and it reports the Sinkhorn-distance analysis showing that the regularizer does minimize distances without improving fairness. That decoupling is a genuinely useful empirical observation. The negative conclusion is also properly hedged: the authors say the results do not support the original claim, not that FairCLIP is harmful.\n\nThe soft spots are real but mechanical. Three seeds per condition is thin, and the paper itself admits that standard deviations are often larger than the differences being compared. So the central negative claim—no improvement in fairness or performance—is consistent with the data but not decisive; it could be a power artifact. The stress-test note is right about that, but not fatal, because the claim is framed as absence of evidence, not evidence of absence. The linear-probe reproduction used 500 epochs instead of the original 1000, which weakens that part. Reusing hyperparameters tuned on gender for other attributes, and building custom prompts for FairFace, further limit the scope. None of these are hidden; they are all in the open.\n\nWho is this for? Anyone working on fairness in vision-language models or reproducibility of CVPR methods. It is the kind of paper that should be in the literature even if the negative result later proves fragile, because the implementation-divergence finding stands on its own. I would cite it for the test-set selection issue alone.\n\nRecommendation: send it to peer review. Ask for more seeds or a power analysis if the venue demands statistical rigor, but the core documentation of implementation divergence and the decoupling result are enough to justify referee time.","headline":"A careful, honest reproduction study that documents real implementation divergences in FairCLIP; the negative result is consistent but underpowered, so treat it as a caution, not a refutation.","tokens_in":27572,"tokens_out":1332,"would_cite":true,"duration_ms":19240,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reproduction study finds that FairCLIP's Sinkhorn-distance fairness regularizer reduces group distance but does not improve CLIP's fairness or accuracy in zero-shot glaucoma classification.","keywords":["FairCLIP","CLIP","group fairness","Sinkhorn distance","reproducibility","zero-shot glaucoma classification","Harvard-FairVLMed","FairFace"],"falsifier":"Run the original FairCLIP code unchanged with model selection on the test set and 1000-epoch linear probes on Harvard-FairVLMed. If FairCLIP then matches or beats CLIP-FT on AUC and DPD as in the original paper, the negative result is caused by the protocol corrections rather than by the method; if it still fails, the method itself does not transfer.","tokens_in":26421,"feed_emoji":"⚖️","tokens_out":4989,"duration_ms":53739,"temperature":0.7,"pith_summary":"This paper tries to establish whether FairCLIP, a method that adds an optimal-transport regularizer to CLIP fine-tuning to reduce group fairness gaps, delivers the fairness and performance gains its authors claimed. Reproducing the original setup on the Harvard-FairVLMed dataset and extending it to FairFace, the authors confirm that CLIP is biased toward certain demographic groups in zero-shot glaucoma classification. They find that FairCLIP's regularization does shrink Sinkhorn distances between population and group similarity-score distributions, but this does not translate into better demographic parity, equalized odds, or AUC; both the official implementation and an implementation aligned with the paper's description fail to beat plain fine-tuned CLIP. The paper also documents that the official code differs from the published model in how similarities are computed and selects models on the test set rather than the validation set, and it introduces FairCLIP+ for multiple sensitive attributes, which shows no significant gains. If the reproduction is faithful, practitioners should not expect FairCLIP fine-tuning to improve fairness or accuracy in these settings.","feed_headline":"FairCLIP fairness gains fail to reproduce","feed_subtitle":"Both official and aligned implementations shrink group distances without improving bias or accuracy in zero-shot glaucoma screening.","key_machinery":"The load-bearing object is the FairCLIP regularizer L_Fair^A = sum_alpha d(D_batch, D_batch^a), where d is the Sinkhorn distance (an entropic optimal-transport cost) between the batch distribution of image-text similarity scores and the distribution for sensitive group alpha. The paper uses this regularizer to ask whether pulling subgroup similarity-score distributions toward the population distribution changes fairness metrics (DPD, DEOdds, ES-AUC) and accuracy (AUC). The reproduction also leans on the distinction between the published formula and the official implementation—normalized group scores, a different similarity matrix, and test-set model selection—with A-FairCLIP as the aligned v","core_discovery":"The central claim under test is that fine-tuning CLIP with the FairCLIP objective improves both performance and fairness of zero-shot glaucoma classification. On both ViT-B/16 and ViT-L/14, the reproduced FairCLIP models generally underperform CLIP-FT on AUC and fairness metrics, often with standard deviations larger than the differences. The official implementation's deviations—normalizing group similarity scores, using diag(I T^T T I^T) instead of diag(I T^T), and selecting the best model on the test set—are identified and corrected in A-FairCLIP, but the aligned models also fail to show consistent improvements. A new multi-attribute variant, FairCLIP+, slightly improves on FairCLIP in som","pith_inferences":["If the original authors' test-set selection is what produced their best scores, then the correct interpretation is that the original evaluation protocol, not the FairCLIP method itself, drove the reported gains; rerunning with held-out selection would settle this.","A stricter test of direction: measure per-group calibration or equalized odds on the exact similarity scores, not just threshold metrics, to see whether distribution matching changes decision boundaries even when AUC does not.","The decoupling of Sinkhorn distance from group fairness may generalize: any regularizer that penalizes only first-order score-distribution overlap could leave rank-ordering biases intact, which matters for other optimal-transport fairness methods.","On heavily imbalanced clinical attributes (ethnicity, language), sample sizes per minority group are tiny, so group-distance estimates are noisy; future reproductions should report uncertainty and consider stratified sampling."],"forward_implications":["FairCLIP and FairCLIP+ do not reliably improve AUC, DPD, DEOdds, or ES-AUC over plain fine-tuned CLIP on Harvard-FairVLMed zero-shot glaucoma classification.","Minimizing Sinkhorn distances between group and population similarity-score distributions is not sufficient for fairness: distances drop while fairness metrics do not.","Fine-tuning CLIP on one sensitive attribute does not consistently give the best fairness on that attribute, and can affect other attributes unpredictably.","On FairFace, the multi-attribute FairCLIP+ objective yields no clear performance improvement and may slightly hurt already-weak age groups.","The original reported gains may be tied to the official pipeline's test-set model selection and unaligned similarity computation, so those parameters should not be transferred to other settings."],"supporting_citations":[{"why":"Original FairCLIP method, claims, datasets, hyperparameters, and the reported results that this study reproduces.","marker":"Luo et al. (2024)"},{"why":"CLIP base model and zero-shot classification setup; supplies the architecture, checkpoints, and contrastive loss being fine-tuned.","marker":"Radford et al. (2021)"},{"why":"Defines the Sinkhorn distance used as the fairness regularizer and as the distance being measured.","marker":"Cuturi (2013)"},{"why":"FairFace dataset used for the out-of-domain generalization experiments.","marker":"Karkkainen & Joo (2021)"},{"why":"Geomloss library used to compute Sinkhorn and MMD distances in the reproduced pipeline.","marker":"Feydy et al. (2019)"},{"why":"Supplies MMD, the alternative distance function tested in place of Sinkhorn.","marker":"Gretton et al. (2012)"},{"why":"Provides the linear probing setup adapted for evaluating the frozen vision encoder's bias.","marker":"He et al. (2022)"}],"fun_headline_variants":["FairCLIP's fairness claim fails to reproduce","Reproducing FairCLIP: no gain in fairness or accuracy","FairCLIP shrinks group gaps but not bias in screening","A-FairCLIP aligns code yet fails to beat CLIP baseline","FairCLIP reproducibility: zero-shot glaucoma sees no benefit"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reproduction protocol—validation-based model selection, aligned similarity scoring, 500-epoch linear probes, and hyperparameters taken from the original authors or tuned on gender alone—is a faithful test of FairCLIP's claimed gains; if the original gains depended on test-set selection or the unaligned scoring, the negative result indicts the original pipeline rather than the method.","fun_headline_variants_meta":{"raw":{"variants":["FairCLIP's fairness claim fails to reproduce","Reproducing FairCLIP: no gain in fairness or accuracy","FairCLIP shrinks group gaps but not bias in screening","A-FairCLIP aligns code yet fails to beat CLIP baseline","FairCLIP reproducibility: zero-shot glaucoma sees no benefit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1297,"prompt_tokens":778,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":522,"tokens_out":519,"duration_ms":6836,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:29:30.187665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original FairCLIP code unchanged with model selection on the test set and 1000-epoch linear probes on Harvard-FairVLMed. If FairCLIP then matches or beats CLIP-FT on AUC and DPD as in the original paper, the negative result is caused by the protocol corrections rather than by the method; if it still fails, the method itself does not transfer.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"CLIP base model and zero-shot classification setup; supplies the architecture, checkpoints, and contrastive loss being fine-tuned."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Defines the Sinkhorn distance used as the fairness regularizer and as the distance being measured."},{"cited_title":"Interpolating between optimal transport and mmd using sinkhorn divergences","cited_arxiv_id":null,"evidence_quote":"Geomloss library used to compute Sinkhorn and MMD distances in the reproduced pipeline."},{"cited_title":"Borgwardt, Malte J","cited_arxiv_id":null,"evidence_quote":"Supplies MMD, the alternative distance function tested in place of Sinkhorn."}],"review_version":1}