{"id":"da3c6197-7fa6-464a-aeeb-46d23416ddbc","arxiv_id":"2505.03153","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A framework adding Dynamic Bad Pair Mining and Sinkhorn distance fairness loss to CLIP and BLIP-2 improves glaucoma diagnosis AUC on Harvard-FairVLMed, but fairness metrics worsen for several protected groups.","lead":"This paper combines two existing training tricks, bad pair mining and a fairness penalty, to make vision-language models for medical images more accurate and fair. It reports improved glaucoma detection AUC on a public dataset, but the fairness gains are inconsistent across demographic groups.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I contradicts the central fairness claim: Robust FairCLIP worsens DPD/DEOdds and ES-AUC for Gender, Ethnicity, and Language, and the 8.6% ES-AUC headline is not supported by the reported ES-AUC column.","rationale":"I agree with the reader's REJECT verdict, but for a different primary reason. The reader's weakest_assumption concerns the Gaussian assumption in DBPM (Section III-C). That is a real weakness, but it is not the most load-bearing issue because the robustness component is never tested experimentally and the paper could partially salvage it by re-running with non-parametric thresholds. The decisive problem is that the fairness claim, which is the title and abstract's central promise, is contradicted by the paper's own Table I on three of the four protected groups. A method that improves overall AUC while worsening DPD/DEOdds or ES-AUC on Gender, Ethnicity, and Language cannot be called a fair VLM framework. The missing 8.6% figure in the ES-AUC column and the absence of error bars/seeds further prevent any quantitative support for the headline. The limitation statement in Section V confirms that only one protected attribute is optimized, making the reported group-wise improvements on other attributes likely incidental. Since the central claim fails on internal evidence, no amount of additional analysis of DBPM would change the verdict; the paper would need a fundamentally revised evaluation and likely a revised method to support 'robust fairness.' Thus the reader's REJECT stands unchanged.","tokens_in":10837,"tokens_out":6511,"duration_ms":55335,"concrete_test":"Reproduce Table I using the Harvard-FairVLMed test split (2,000 samples) and the exact training setup in Section IV-A (CLIP ViT-B/16, Adam, lr=1e-5, batch size 32, 10 epochs, epsilon=1e-4, lambda=1e-7, alpha=beta=3). Compute DPD, DEOdds, and ES-AUC separately for Race, Gender, Ethnicity, and Language. If Gender DPD/DEOdds, Ethnicity DPD/DEOdds/ES-AUC, and Language DEOdds/ES-AUC reproduce the degradations in Table I, then the fairness claim fails on the paper's own metric definitions. If the code link is repaired and the numbers differ, the reported tables are unreliable; either way the central fairness conclusion is not supported as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the paper's own evaluation refutes its central fairness claim. In Table I, relative to FairCLIP, Robust FairCLIP degrades fairness on protected attributes: Gender DPD 0.35->2.71 and DEOdds 5.54->7.08; Ethnicity DPD 5.35->9.85, DEOdds 8.88->14.00, and ES-AUC 64.1->61.71; Language DEOdds 16.3->23.77 and ES-AUC 59.5->58.59. The abstract claims 'up to a 8.6% improvement when looking at equity-scaled AUC', but the ES-AUC column in Table I changes from 56.76 (CLIP) to 65.88 (Robust FairCLIP), a 16.1% relative change, and from 65.7 (FairCLIP) to 65.88, a 0.27% relative change; no reported subgroup ES-AUC value yields 8.6%. The only consistent gain is overall AUC (67.6->70.84 for CLIP, 80.03->81.96 for BLIP-2), which is an accuracy improvement, not fairness. The Section V limitation admits the fairness loss optimizes only one protected attribute at a time, which is consistent with the degradation on the other attributes; yet the paper presents all four attributes as successes. Thus the central claim that the framework 'ensures fairness' is contradicted by the paper's own tables.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for robust and fair vision-language learning in medical image analysis. It combines a Dynamic Bad Pair Mining (DBPM) algorithm, adapted from time-series contrastive learning, with a Sinkhorn-distance-based fairness regularizer inspired by FairCLIP, and applies this to fine-tune CLIP ViT-B/16 and BLIP-2 on the Harvard-FairVLMed glaucoma dataset. The authors report overall AUC improvements over the CLIP and FairCLIP baselines and claim up to an 8.6% improvement in equity-scaled AUC, concluding that the framework ensures robustness and fairness for VLMs in medical diagnosis.","tokens_in":11175,"tokens_out":4226,"duration_ms":40499,"significance":"The problem is timely and important: medical VLMs need to be robust to noisy text-image pairs and fair across demographic groups. The paper has useful ingredients: it uses a public benchmark, compares against two relevant baselines, tests two backbone architectures, and releases code. The overall AUC gains (67.6 to 70.84 for CLIP; 80.03 to 81.96 for BLIP-2) are nontrivial if reproducible. However, the central fairness claim is contradicted by the paper's own reported metrics, and the robustness contribution is not supported by ablations, uncertainty quantification, or a methodological justification of the batch-level Gaussian assumption. As it stands, the paper establishes at most an accuracy improvement from a particular combination of existing losses, not a robust and fair framework.","major_comments":[{"comment":"Table I contradicts the central fairness claim. Relative to FairCLIP, Robust FairCLIP worsens Gender DPD from 0.35 to 2.71 and DEOdds from 5.54 to 7.08; Ethnicity DPD from 5.35 to 9.85, DEOdds from 8.88 to 14.00, and ES-AUC from 64.1 to 61.71; Language DEOdds from 16.3 to 23.77 and ES-AUC from 59.5 to 58.59. The only consistent gains are overall AUC and some group-wise AUCs, which are accuracy improvements, not fairness improvements. Moreover, the abstract's claim of \"up to a 8.6% improvement when looking at equity-scaled AUC\" is not supported by the ES-AUC column: the overall ES-AUC changes from 56.76 (CLIP) to 65.88 (Robust FairCLIP), a relative change of 16.1%, and from 65.7 (FairCLIP) to 65.88, a change of only 0.27%; no subgroup ES-AUC value in Table I yields 8.6%. Section V's own limitation statement admits that the fairness loss optimizes only one protected attribute at a time, which is consistent with the observed degradation on other attributes, yet the paper presents all four attributes as successes. This directly undermines the claim that the framework \"ensures fairness.\"","section":"Table I and Section V (Limitation)"},{"comment":"No uncertainty quantification is reported. All results are single-run point estimates without error bars or multiple seeds, and several protected groups are very small (for example, Spanish speakers are 1.7% and users of other languages 0.8% of the dataset). Differences in DPD and DEOdds of a few percentage points may be within noise, so the claimed improvements over FairCLIP and BLIP-2 cannot be assessed reliably. The authors should report means and standard deviations over at least 3-5 seeds and, where feasible, confidence intervals for the fairness metrics.","section":"Section IV-A (Experimental Settings) and Tables I-II"},{"comment":"The DBPM component operates at the batch level, not at the individual image-text pair level, which contradicts the claim that it identifies \"faulty image-text pairs.\" The historical loss S(i,e) is defined per batch, and the weights w(i,e) down-weight entire batches whose mean loss falls in the tails. A high-loss batch could contain many correct pairs and one very faulty pair, or vice versa; batch-level weighting cannot distinguish these cases. In addition, the Gaussian assumption for the distribution of batch losses is asserted without evidence or a goodness-of-fit check, so the thresholds in Eq. (2) are not justified. An ablation isolating the DBPM contribution and a sensitivity analysis over alpha and beta are needed.","section":"Section III-C (Dynamic Bad Pair Mining)"},{"comment":"The fairness term is not shown to control the reported fairness metrics. Eq. (6) adds a Sinkhorn distance between the distribution of positive-pair similarities for a protected group and the overall batch distribution, while the evaluation uses DPD and DEOdds computed from final predictions. No derivation or empirical analysis links this regularizer to the measured fairness metrics. Since the reported results show degraded fairness on several attributes, the choice of lambda = 1e-7 and the single-attribute form of the loss require justification and ablation studies.","section":"Section III-D and Eq. (6)"}],"minor_comments":[{"comment":"The abstract attributes the 8.6% gain to equity-scaled AUC, while Section IV-B states the improvement is over total AUC; these statements are inconsistent and should be reconciled.","section":"Abstract and Section IV-B"},{"comment":"The notation is inconsistent: the dataset is defined as D = {x_i, y_i, d_i, p_i} with d_i as diagnosis and p_i as protected attributes, but later gamma_i is introduced for protected attributes, and Eq. (1) uses d_i as a scalar label inside the log term. Please standardize the notation.","section":"Section III-B"},{"comment":"Eq. (4) is typeset incorrectly: the summation over gamma and the distance function are garbled, and the objective as printed is not a valid optimization statement.","section":"Section III-D, Eq. (4)"},{"comment":"The code link contains a space (\"Robust Fairness for Medical Image.git\") and is not a valid URL.","section":"Abstract"},{"comment":"There are numerous typos, including \"Dymanic\" in Section II-B, \"a issue\" in the Introduction, and \"LLaV A\" in Section II-A.","section":"Throughout"},{"comment":"The caption states that bolded values represent the best value in the group, but Robust FairCLIP's worse DPD and DEOdds values are not bolded; please verify the bolding convention.","section":"Table I caption"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a straightforward combination of two existing losses: Dynamic Bad Pair Mining (DBPM) from time-series contrastive learning, and the Sinkhorn fairness loss from FairCLIP. The application to CLIP and BLIP-2 on Harvard-FairVLMed is new but incremental. Second, the paper's own numbers undercut its main claim: it promises robust fairness, but in Table I the fairness metrics worsen for Gender, Ethnicity, and Language relative to FairCLIP. The Race row genuinely improves, and overall AUC goes up, but that is an accuracy gain, not a fairness gain.\n\nWhat works: real experiment, real medical fairness dataset, two architectures, group-wise AUC alongside DPD/DEOdds. The Race result - DPD 14.1 to 11.05, DEOdds 17.8 to 11.16, all group AUCs up - is worth taking seriously. The limitation section is candid that only one protected attribute is explicitly optimized, though that candor creates a problem below.\n\nSoft spots:\n\n(1) The 8.6% equity-scaled AUC improvement does not appear in the tables. ES-AUC moves 56.76 to 65.88 versus CLIP (16% relative, not 8.6%) and 65.7 to 65.88 versus FairCLIP (flat). No subgroup value yields 8.6%. If it comes from another run, say where.\n\n(2) Robustness is never tested. No injected noise, corrupted text, or label perturbation. DBPM is added, but the evaluation only reports AUC and fairness.\n\n(3) No error bars or repeated seeds; one run per model.\n\n(4) The code link has spaces and doesn't resolve.\n\n(5) DBPM weights whole batches, not individual pairs, and the Gaussian weight is an unnormalized PDF - it can exceed 1 when sigma is small.\n\n(6) Equation (6) sums the Sinkhorn penalty over all protected attributes, but the limitation says only one is optimized. If each row is a separate model, cross-attribute comparisons are misleading; if one model, the limitation is wrong. The paper should clarify.\n\nThe citation list leans heavily on the authors' own group; not fatal, but noticeable.\n\nBottom line: this deserves peer review, not desk rejection - the combination is real and the dataset is appropriate. The audience is people working on fair medical VLMs. I would send it out with a request for major revision: corrected headline, robustness experiments, error bars, clarification of which model produced which row, and a discussion of why fairness degrades on three of four attributes. I wouldn't cite it in its current form.","headline":"A competent but overclaimed combination of two existing losses: the fairness claim is contradicted by the paper's own Table I, and the 8.6% headline is nowhere in the data.","tokens_in":11684,"tokens_out":10276,"would_cite":false,"duration_ms":83977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training framework combines bad-pair mining with a Sinkhorn-distance fairness loss, improving equity-scaled AUC of medical vision-language models by up to 8.6%.","keywords":["fairness","vision-language models","medical image analysis","contrastive learning","robustness","Sinkhorn distance","bad pair mining","glaucoma detection"],"falsifier":"Train the same model on a dataset whose batch losses are known to be strongly skewed or bimodal, for example by injecting a systematic proportion of corrupted image-text pairs, then check whether DBPM's down-weighting still improves equity-scaled AUC; if it does no better than randomly down-weighting the same fraction of batches, the Gaussian-threshold mechanism is not the cause. A simpler check is to compute a normality statistic on the recorded historical batch losses during training, since strong evidence against normality would undermine the threshold derivation.","tokens_in":10637,"feed_emoji":"🩺","tokens_out":9636,"duration_ms":85630,"temperature":0.7,"pith_summary":"This paper proposes a training framework that makes vision-language models for medical diagnosis both more accurate and fairer. The authors argue that two problems compound in medical VLMs: clinical notes summarized by large language models create noisy or faulty image-text pairs, and the models inherit demographic biases from pretraining. Their framework attacks both by modifying the contrastive loss during fine-tuning. A Dynamic Bad Pair Mining algorithm identifies and down-weights high- and low-loss batches, and a Sinkhorn-distance term penalizes divergence between the loss distribution of each protected group and the overall batch. On the FairVLMed glaucoma dataset, the resulting models outperform both the CLIP baseline and FairCLIP, with up to an 8.6% gain in equity-scaled AUC, a metric that rewards overall accuracy and penalizes gaps between demographic groups.","feed_headline":"Medical vision-language models gain 8.6% in equity-scaled AUC","feed_subtitle":"Bad-pair mining plus Sinkhorn-distance fairness narrows gaps between demographic groups while raising accuracy.","key_machinery":"The load-bearing mechanism is the interaction of two loss components. Dynamic Bad Pair Mining, adapted from time-series contrastive learning, estimates the mean $\\mu_e$ and standard deviation $\\sigma_e$ of historical per-batch losses and defines an interval $[a,b]$; batches outside it are down-weighted by a Gaussian reduction weight instead of removed. The Sinkhorn-distance fairness term $S_\\epsilon(B_W,B_{W_\\gamma})$ is an entropy-regularized optimal transport cost between the batch's loss distribution and each protected group's distribution, added with regularization coefficient $\\lambda$. These two terms are combined into a final loss $L_3 = L_2 + \\lambda \\sum_\\gamma S_\\epsilon(B_W,B_{W_\\gamma})$, where $L_2$ is the bad-pair-weighted contrastive loss.","core_discovery":"The central claim is that robustness and fairness can be added to any feature-based medical vision-language model through two loss-side modifications, without changing the backbone architecture. Training starts from the standard CLIP contrastive objective using symmetric cross-entropy. The framework records each batch's historical loss, assumes these per-batch losses are roughly Gaussian, and uses thresholds $a = \\mu - \\alpha \\sigma$ and $b = \\mu + \\beta \\sigma$ to mark batches as correct, noisy, or faulty; noisy and faulty batches receive reduced weights from a Gaussian kernel. It then adds a Sinkhorn-distance regularizer that measures, for each protected attribute such as race or gender, how far that group's similarity-score distribution sits from the overall batch distribution. The paper reports that this combined loss raises overall AUC (area under the ROC curve) from 63.68 to 70.84 for a CLIP ViT-B/16 backbone and improves equity-scaled AUC from 56.76 to 65.88, with similar gains when the same framework is applied to BLIP-2.","pith_inferences":["Beyond the paper, the Gaussian assumption behind DBPM is directly testable: a goodness-of-fit test on the historical loss distribution during training would show whether the thresholds select noise or simply truncate a skewed distribution.","Beyond the paper, the fairness regularizer is currently applied per protected attribute separately; a joint or intersectional Sinkhorn term might close the remaining ethnicity and language gaps visible in the reported group-wise AUCs.","Beyond the paper, since DBPM weights whole batches rather than individual image-text pairs, a pair-level noisy-label model could be more precise; comparing the two on the same dataset would quantify how much granularity matters.","Beyond the paper, one could test the framework on a second disease or imaging modality; if the gains replicate, the combination of bad-pair down-weighting and distributional fairness is a general recipe for medical VLMs, not a glaucoma-specific fix."],"forward_implications":["If the framework is correct, medical VLMs can be fine-tuned on large, noisy datasets without a fairness/accuracy trade-off: the reported numbers show accuracy and equity improving together.","Because the same loss modifications improved both CLIP and BLIP-2, the method is architecture-agnostic and can be bolted onto future feature-based VLMs.","The Sinkhorn regularizer visibly narrows group-wise AUC gaps for race and ethnicity, suggesting a route to more equitable glaucoma screening.","DBPM's batch-level down-weighting makes training more stable when clinical notes are LLM-summarized, removing the need for manual pair cleaning.","The framework's reported gains are measured by equity-scaled AUC, meaning improvements are counted only insofar as they hold across all protected groups."],"supporting_citations":[{"why":"Supplies the CLIP model and the contrastive image-text objective that the framework modifies.","marker":"[16]"},{"why":"Provides the FairCLIP baseline, the Sinkhorn fairness loss, and the FairVLMed dataset used in experiments.","marker":"[18]"},{"why":"Supplies the Dynamic Bad Pair Mining algorithm that the paper adapts from single-modal to multimodal training.","marker":"[43]"},{"why":"Provides the BLIP-2 architecture used to test whether the framework transfers across model families.","marker":"[44]"},{"why":"Defines the symmetric cross-entropy loss used as the base contrastive loss in equation (1).","marker":"[55]"},{"why":"Supplies the Sinkhorn distance / entropy-regularized optimal transport formulation used in the fairness loss.","marker":"[56]"},{"why":"Defines the equity-scaled AUC metric used to evaluate performance across protected groups.","marker":"[57]"}],"fun_headline_variants":["Loss tweaks lift fairness in medical VLMs by 8.6%","Bad-pair mining plus Sinkhorn distance aids medical VLM equity","Medical VLM fairness gains 8.6% via twin loss adjustments","Dynamic mining and Sinkhorn regularizer boost medical VLM AUC","Fairer medical vision-language models via loss-side fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the historical per-batch losses are Gaussian, so that the mean-and-standard-deviation thresholds $a$ and $b$ separate noisy batches from clean ones; if the loss distribution is skewed or bimodal, the down-weighting becomes arbitrary, and the method also operates on whole batches rather than individual image-text pairs.","fun_headline_variants_meta":{"raw":{"variants":["Loss tweaks lift fairness in medical VLMs by 8.6%","Bad-pair mining plus Sinkhorn distance aids medical VLM equity","Medical VLM fairness gains 8.6% via twin loss adjustments","Dynamic mining and Sinkhorn regularizer boost medical VLM AUC","Fairer medical vision-language models via loss-side fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1220,"prompt_tokens":883,"completion_tokens":337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":245}},"tokens_in":499,"tokens_out":337,"duration_ms":3777,"temperature":1.0,"reasoning_tokens":245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:58:02.428056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on a dataset whose batch losses are known to be strongly skewed or bimodal, for example by injecting a systematic proportion of corrupted image-text pairs, then check whether DBPM's down-weighting still improves equity-scaled AUC; if it does no better than randomly down-weighting the same fraction of batches, the Gaussian-threshold mechanism is not the cause. A simpler check is to compute a normality statistic on the recorded historical batch losses during training, since strong evidence against normality would undermine the threshold derivation.","supporting_citations":[{"cited_title":"Harvard glaucoma fairness: a retinal nerve disease dataset for fairness learning and fair identity normalization,","cited_arxiv_id":null,"evidence_quote":"Defines the equity-scaled AUC metric used to evaluate performance across protected groups."},{"cited_title":"Computational optimal transport: With applications to data science,","cited_arxiv_id":null,"evidence_quote":"Supplies the Sinkhorn distance / entropy-regularized optimal transport formulation used in the fairness loss."},{"cited_title":"Fairclip: Harnessing fairness in vision-language learning,","cited_arxiv_id":null,"evidence_quote":"Provides the FairCLIP baseline, the Sinkhorn fairness loss, and the FairVLMed dataset used in experiments."},{"cited_title":"Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach","cited_arxiv_id":"2302.03357","evidence_quote":"Supplies the Dynamic Bad Pair Mining algorithm that the paper adapts from single-modal to multimodal training."},{"cited_title":"Symmetric cross entropy for robust learning with noisy labels,","cited_arxiv_id":null,"evidence_quote":"Defines the symmetric cross-entropy loss used as the base contrastive loss in equation (1)."}],"review_version":1}