{"id":"0fe0e052-e82f-49f5-9d26-f3f01aa0ed3d","arxiv_id":"2502.06094","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Fair-MoE reports improved accuracy and fairness on Harvard-FairVLMed for some protected attributes by adding sparse mixture-of-experts layers and a variance-based fairness loss to CLIP, but the all-attribute improvement claim is contradicted by its own tables.","lead":"This paper introduces Fair-MoE, a vision-language model that adds mixture-of-experts layers and a fairness loss to CLIP, aiming to reduce demographic bias in medical image diagnosis. Tested on a glaucoma image-text dataset, the method reports better accuracy and fairness than two baselines for some attributes, but the paper's own results contradict its headline claim of improvement across all four protected attributes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 directly contradicts the abstract's \"all four attributes\" claim: Fair-MoE/ViT-L14 has worse DPD and EOD than FairCLIP for gender and language, so the paper's central empirical claim fails by its own numbers.","rationale":"The reader's REJECT verdict is well grounded. My stress-test focuses on what is, in my view, the most load-bearing element: the paper's central claim of fairness gains across all four attributes is falsified by its own Table 1. This is not a matter of theoretical disagreement or external consensus; it is an internal contradiction. The abstract says 'improvements in both fairness and accuracy across all four attributes,' yet for ViT-L14 the model is worse than FairCLIP on DPD and EOD for gender and language. The paper's text tries to narrow the EOD claim to three attributes, but even the DPD claim is contradicted by the language row. This single issue is sufficient to reject the central empirical claim. The reader highlighted a different weakest assumption (that gate-weight variance is a meaningful fairness proxy). That concern is real and also load-bearing: the ablations in Tables 4 to 6 report only ES-AUC, so they never show whether the proposed FOL actually improves DPD/EOD. However, the overclaim is more directly decisive, because the paper's own numbers refute the headline. I therefore set agreement_with_reader to partial: I endorse the REJECT verdict and many of the reader's observations, but my primary concern is the internal contradiction rather than the mechanism-level proxy validity. The concrete test is deliberately simple: re-derive the comparisons from Table 1 and re-run ablations with fairness metrics. Because the paper provides no code or hyperparameters, this verification would require the authors to supply the missing details; without that, the reported numbers cannot be checked independently. Fair-MoE's underlying idea remains worth studying, but the current write-up does not support its claimed results.","tokens_in":14531,"tokens_out":2988,"duration_ms":25764,"concrete_test":"Independently compare Table 1 row by row: for each protected attribute and each architecture (ViT-B16 and ViT-L14), determine whether Fair-MoE strictly improves DPD and EOD relative to both CLIP and FairCLIP. If any attribute shows worse DPD or EOD than FairCLIP (as appears for gender and language under ViT-L14), the abstract's 'all four attributes' claim is false. Additionally, rerun the ablations in Tables 4 to 6 and report DPD and EOD alongside ES-AUC; if removing FOL components does not significantly change DPD/EOD, the claim that gate-variance dispersion is indispensable for fairness is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, repeated in the abstract and in Section 3.2, is that Fair-MoE improves fairness and accuracy across all four protected attributes and that FOL's gate-variance term is indispensable. The paper's own Table 1 does not support this. For ViT-L14, Fair-MoE/l14 reports DPD 15.67±2.99 and EOD 23.63±14.40 for language, versus FairCLIP/l14's DPD 8.21±1.99 and EOD 11.00±1.25; for gender, Fair-MoE/l14 reports DPD 2.94±1.60 and EOD 7.33±2.55, versus FairCLIP/l14's DPD 2.11±1.81 and EOD 5.24±1.46. Fair-MoE is therefore worse on both fairness metrics for two of four attributes in the larger architecture. The paper's own narrative partially concedes this by claiming EOD gains only for race, gender, and ethnicity, while the abstract asserts gains for all four. The DPD claim in Section 4.2 that Fair-MoE 'achieves better fairness than baselines in all attributes' is also false for language under ViT-L14. Additionally, the ablation studies central to the FOL indispensability argument (Tables 4 to 6) report only ES-AUC, not DPD/EOD, so they provide no direct evidence that the variance term improves demographic parity or equalized odds. Thus the load-bearing empirical assertion is internally contradicted, and the mechanism-level claim is not assessed with outcome fairness metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Fair-MoE, a vision-language model for medical image analysis that replaces the final MLP layers in CLIP with a mixture-of-experts architecture (FO-MoE) and adds a fairness-oriented loss (FOL) that combines Sinkhorn distance with a penalty on the variance difference of gate weights between protected attribute groups. The authors claim that Fair-MoE improves both accuracy (AUC) and fairness (DPD, EOD, ES-AUC) over CLIP and FairCLIP across all four protected attributes (race, gender, ethnicity, language) on the Harvard-FairVLMed dataset. The paper includes a main results table, ablation studies for FO-MoE and FOL, and component-wise ablations for the four variance-based loss terms.","tokens_in":14906,"tokens_out":5246,"duration_ms":41801,"significance":"If the empirical claims held, Fair-MoE would offer a simple and general recipe for debiasing medical VLMs: route patches and features through a sparse mixture of experts and regularize the gate-weight variance across protected groups. The idea of using gate-weight dispersion as a fairness signal is interesting, and the ablation structure is comprehensive in intent. However, the headline claim is contradicted by the paper's own main results table, and the fairness metric definitions are flawed. As a result, the significance of the contribution as stated is not established; the paper needs a corrected set of experiments and metric definitions before its claims can be assessed.","major_comments":[{"comment":"The abstract claims 'improvements in both fairness and accuracy across all four attributes,' but Table 1 contradicts this. For ViT-L14 under Language, Fair-MoE/l14 reports DPD 15.67±2.99 and EOD 23.63±14.40 versus FairCLIP/l14's DPD 8.21±1.99 and EOD 11.00±1.25; for Gender, Fair-MoE/l14 reports DPD 2.94±1.60 and EOD 7.33±2.55 versus FairCLIP/l14's DPD 2.11±1.81 and EOD 5.24±1.46. Fair-MoE is therefore worse on both fairness metrics for two of four attributes in the larger architecture. The sentence in §4.2 stating that 'results of DPD show that Fair-MoE achieves better fairness than baselines in all attributes' is false for Language under ViT-L14, and the same paragraph already concedes that EOD gains hold only for race, gender, and ethnicity. The paper's central empirical claim is internally contradicted by its own numbers.","section":"§4.2, Table 1"},{"comment":"The Demographic Parity Difference is defined as DPD_s = |max_a P(ŷ=1|G=a, y=1) − min_b P(ŷ=1|G=b, y=1)|, which conditions on the true label y=1. This is not demographic parity; it is the true-positive-rate gap, which is the first term of the Equalized Odds Difference defined immediately afterwards. Thus DPD and EOD are not independent fairness metrics as reported. All fairness claims in the abstract, §4.2, and the conclusions that refer to 'demographic parity' are therefore unsubstantiated, and the numerical DPD values would need to be recomputed under the correct conditional probability P(ŷ=1|G=a).","section":"§4.1, metric definitions"},{"comment":"The ablation study does not support the claim that FOL is indispensable for fairness. Table 3 shows several cases where removing FOL improves DPD or EOD: FairMoE/b16 race DPD worsens from 3.19 (without FOL) to 7.25 (with FOL); FairMoE/b16 ethnicity DPD worsens from 6.94 to 8.52; FairMoE/l14 language DPD worsens from 10.43 to 15.67 and EOD from 9.65 to 23.63. The text states that removing FOL makes 'metrics that measure effectiveness and fairness deteriorate significantly,' which is not uniformly true. Moreover, Tables 4–6, used to argue the necessity of the four variance-based loss components, report only ES-AUC and never report DPD or EOD, so they cannot establish that these components improve outcome-level fairness metrics.","section":"§4.3, Tables 3–6"},{"comment":"The proposed mechanism is under-validated. Section 3.1 asserts that the capacity parameter C 'filter[s] out biased path embeddings,' but no analysis is provided of which patches are actually dropped or whether the top-k capacity preferentially discards biased rather than arbitrary low-weight embeddings. Section 3.2 treats the variance of gate weights as a proxy for fairness, but the paper gives no empirical or theoretical connection between gate-weight variance and the outcome-level metrics DPD/EOD. Because the FOL losses are computed on gate weights rather than on predictions, the claimed fairness gains are not attributable to the proposed mechanism without such a link.","section":"§3.1–3.2"}],"minor_comments":[{"comment":"The notation 'T opc' and 'T opr' should be typeset as proper functions, and the capacity formula α = C(N+1)k1/M1 needs a clearer derivation; as printed, the expression is difficult to parse.","section":"§3.1"},{"comment":"The experimental setup does not report the key hyperparameters: number of experts M1 and M2, top-k values k1 and k2, capacity α, the sample size N for variance estimation, or the loss weights of FEI, FET, FFI, FFT, and Ldistance. Without these, the experiments cannot be reproduced.","section":"§4.1"},{"comment":"Several sentences need rewriting for clarity, for example 'Utilizing FO-MoE achieves higher AUC for all attributes demonstrates and gains 1.1% improvements in race' is not grammatical and obscures the intended claim.","section":"§4.3"},{"comment":"The figures are referenced but the caption text does not fully explain the color coding or the data-flow steps in FO-MoE; please ensure the captions are self-contained so the reader can interpret the proposed architecture without the main text.","section":"Figure 1 and Figure 2"},{"comment":"The caption says 'green text highlights our method,' but the table as rendered does not use green highlighting; please fix the formatting or remove the caption statement.","section":"Table 1"},{"comment":"There is a typo: 'textural data' should be 'textual data'.","section":"Abstract/Section 1"},{"comment":"The statement that Fair-MoE 'outperforms baselines 5.00% in ES AUC' for race is ambiguous: please specify the baseline and whether the number is an absolute difference in ES-AUC points.","section":"§4.2"},{"comment":"The statement 'Code will be publicly available' should include a URL or a planned release identifier, as the current wording is not actionable.","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The paper's primary claim is contradicted by its own Table 1, and the DPD definition is mathematically wrong, so the current manuscript does not meet the bar for publication. In addition, the absence of hyperparameter reporting and the inconsistent ablation results make it difficult to see how a revision within the normal scope could salvage the central claims without a substantial new empirical study. The naming overlap with Germino et al.'s 'FairMoE' is worth noting to the editor, though the authors do cite that work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fair-MoE is a reasonable idea: add sparse MoE routing to CLIP encoders and regularize with a loss that penalizes group differences in gate-weight variance. That is new for medical VLMs, and the ablations in Tables 4–6 suggest the components matter for the ES-AUC trade-off. The paper is clearly written and the related work is competent.\n\nBut the headline claim does not survive contact with its own numbers. The abstract says improvements in fairness and accuracy 'across all four attributes.' Table 1 shows the ViT-L14 variant is worse than FairCLIP on both DPD and EOD for language (15.67 vs 8.21; 23.63 vs 11.00) and for gender (2.94 vs 2.11; 7.33 vs 5.24). Section 4.2's statement that DPD is better in all attributes is false for language. Ablation Table 3 tells the same story: adding FOL makes language fairness worse in the l14 variant (DPD 15.67 vs 10.43, EOD 23.63 vs 9.65). So the claim that FOL is indispensable is not supported for a third of the attributes in the larger architecture.\n\nThere are other problems. The DPD definition given in Section 4.1 conditions on y=1, which is not demographic parity — it is part of equalized odds. That is a metric error, not just a notation quirk. The reported ES-AUC for FairMoE/b16 race is 69.63 in Table 1 but 70.9 in Tables 4–6; the l14 row changes too. No code, no hyperparameters, and the 'biased patch embedding filtering' mechanism is asserted but never analyzed — no examples of dropped tokens, no link between gate variance and outcome parity. The reproducibility of the whole thing is not yet there.\n\nThe core idea — using gate statistics as a fairness handle — is worth taking seriously. A rigorous version with corrected metrics, consistent tables, code, and an honest account of where the method fails could be a useful contribution. This version isn't there yet.\n\nI'd send it to reviewers if an editor wanted a second opinion, because the problem is important and the design is interesting, but I would expect the referee reports to be harsh. For my own work, I wouldn't cite it until the results are corrected and reproduced.","headline":"The central claim — improvements across all four attributes — is contradicted by the paper's own Table 1; the MoE-for-fairness idea is plausible but this version overclaims and has reproducibility gaps.","tokens_in":15491,"tokens_out":2911,"would_cite":false,"duration_ms":23386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mixture-of-experts layer plus a variance-matching fairness loss improves both accuracy and fairness of medical vision-language models on a glaucoma benchmark.","keywords":["fairness","vision-language models","mixture of experts","medical image analysis","contrastive learning","Sinkhorn distance","glaucoma","demographic parity"],"falsifier":"Log the per-group variance of gate weights alongside DPD and EOD throughout training of Fair-MoE; if the fairness metrics improve without the variance-difference term shrinking, or the variance differences shrink while DPD and EOD stay flat, the claim that gate-weight dispersion drives fairness is refuted, and the same logging can reveal whether the capacity filter's dropped patches correlate with protected attributes.","tokens_in":14244,"feed_emoji":"⚖️","tokens_out":13723,"duration_ms":99754,"temperature":0.7,"pith_summary":"This paper proposes Fair-MoE, a vision-language model for medical image diagnosis that aims to improve diagnostic accuracy and demographic fairness at the same time. Tested on a glaucoma image-and-text dataset, the model reports better area under the ROC curve and better fairness metrics—demographic parity difference, equal opportunity difference, and equity-scaled AUC—than a standard contrastive vision-language model and a prior fairness-oriented version of it, across all four protected attributes: race, gender, ethnicity, and language. The authors claim the gains come from two design choices: replacing the last-layer MLPs with a sparse mixture of experts that can filter out biased patch embeddings, and adding a fairness loss that matches both the mean and the variance of gating weights between protected groups. If the reported results hold, the approach would give medical VLMs a way to reduce demographic disparities without adding parameters or sacrificing accuracy.","feed_headline":"Fair-MoE lifts both fairness and accuracy for medical VLMs","feed_subtitle":"Sparse experts plus variance-matching loss lifts glaucoma accuracy for all protected groups","key_machinery":"The machinery has two parts. The Fairness-Oriented Mixture of Experts (FO-MoE) replaces the MLP in the final attention block with an embedding-based MoE and places a second feature-based MoE after the encoder; each uses multiple MLP experts and a gating network, keeps only the top-k expert weights per input, and, in the embedding-based layer, caps how many patch embeddings each expert may process (a capacity filter) so that supposedly biased, task-irrelevant patch embeddings are dropped. The Fairness-Oriented Loss (FOL) adds to the standard Sinkhorn distance loss four terms that each compute, for one of the four MoE modules, the squared difference between the variance of expert gate weights over the whole dataset and the variance over a protected group, summed over groups and experts. FOL is thus a load-balancing loss that also acts as a fairness regularizer by making the dispersion of expert usage similar across demographic groups.","core_discovery":"The central claim is that a fairness-oriented mixture-of-experts layer plus a loss that penalizes differences in the dispersion of gating weights across protected groups simultaneously improves effectiveness and fairness in medical vision-language models. Concretely, the paper reports that Fair-MoE with a ViT-L/14 encoder reaches an equity-scaled AUC of 72.53 on the race attribute compared with 67.53 for the FairCLIP baseline, with demographic parity difference dropping from 16.01 to 2.63, while overall AUC rises from 71.57 to 73.93; similar patterns appear for gender, ethnicity, and language. While the baseline FairCLIP regularizes only the Sinkhorn distance between group distributions, Fair-MoE adds variance-of-gate-weight matching, and the ablations show that removing this variance term degrades both accuracy and fairness. The paper also shows that both the embedding-level and feature-level expert layers contribute to the reported trade-off.","pith_inferences":["The paper does not show that gate-weight variance tracks the demographic parity or equal-opportunity disparities it optimizes; a natural next step would be to measure that correlation during training and test whether a control loss of equal strength on randomly shuffled group labels reproduces the fairness gains.","A testable extension is to apply the same FO-MoE-plus-FOL design to other medical imaging tasks and check whether the fairness gains generalize beyond glaucoma or depend on the particular group sizes in the benchmark dataset.","Because the capacity filter is not analyzed in terms of which patches are dropped, the stated bias-filtering mechanism could be checked by comparing the distribution of dropped patch locations with protected-attribute content; if the dropped patches do not correlate with protected content, the effect may come from regularization rather than targeted filtering.","If variance matching is the operative ingredient, connecting FOL to a second-moment moment-matching or optimal-transport constraint could give a principled account of when the loss helps and when it might conflict with task accuracy."],"forward_implications":["If the reported results are correct, fairness gains in medical VLMs do not require retraining from scratch or adding parameters: the paper states parameter counts stay around 200M for the smaller encoder and 500M for the larger one.","The ablation results imply that matching the variance of gate weights between groups is load-bearing; removing it measurably lowers both AUC and equity-scaled AUC.","Because both the image-branch and text-branch MoE ablations hurt performance when removed, the paper's account predicts that debiasing a multimodal medical VLM requires intervening in both modalities, not only on the visual side.","The capacity filter is presented as the mechanism that discards biased patch embeddings; if that is true, the same sparse-expert recipe could be transferred to other contrastive VLMs to remove spurious correlates before they enter the task representation."],"supporting_citations":[{"why":"Supplies the benchmark glaucoma dataset, the fairness-regularized baseline, and the evaluation protocol and metrics.","marker":"[Luo et al., 2024]"},{"why":"Provides the Sinkhorn distance that forms the distance component of the FOL loss.","marker":"[Peyré et al., 2019]"},{"why":"Introduces sparse mixture-of-experts with top-k gating, which the FO-MoE adapts for fairness.","marker":"[Riquelme et al., 2021]"},{"why":"Provides the load-balancing-by-variance idea that FOL repurposes into a fairness loss.","marker":"[Lou et al., 2021]"},{"why":"Defines the vision transformer structure whose last attention block and class token FO-MoE modifies.","marker":"[Dosovitskiy et al., 2021]"},{"why":"Introduces the mixture-of-experts gating formulation that the paper's expert weighting scheme builds on.","marker":"[Jacobs et al., 1991]"}],"fun_headline_variants":["Fair-MoE boosts fairness and accuracy for medical VLMs","Variance-matching expert loss improves medical VLM equity","Equity-scaled AUC leaps with Fair-MoE","Demographic parity difference drops from 16 to 2.6","Sparse experts plus variance loss: fairer medical AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variance of expert-gate weights across a protected group is a meaningful stand-in for the demographic disparities that the fairness metrics measure: if evening out gate-weight variance does not actually track or cause equal outcome rates, the proposed loss is optimizing an irrelevant quantity and the reported fairness improvements would not be attributable to the mechanism claimed.","fun_headline_variants_meta":{"raw":{"variants":["Fair-MoE boosts fairness and accuracy for medical VLMs","Variance-matching expert loss improves medical VLM equity","Equity-scaled AUC leaps with Fair-MoE","Demographic parity difference drops from 16 to 2.6","Sparse experts plus variance loss: fairer medical AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3526,"prompt_tokens":986,"completion_tokens":2540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2455}},"tokens_in":602,"tokens_out":2540,"duration_ms":18452,"temperature":1.0,"reasoning_tokens":2455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:48:00.493077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Log the per-group variance of gate weights alongside DPD and EOD throughout training of Fair-MoE; if the fairness metrics improve without the variance-difference term shrinking, or the variance differences shrink while DPD and EOD stay flat, the claim that gate-weight dispersion drives fairness is refuted, and the same logging can reveal whether the capacity filter's dropped patches correlate with protected attributes.","supporting_citations":[{"cited_title":"Fairclip: Harnessing fairness in vision-language learning","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark glaucoma dataset, the fairness-regularized baseline, and the evaluation protocol and metrics."},{"cited_title":"Scaling vision with sparse mixture of experts","cited_arxiv_id":null,"evidence_quote":"Introduces sparse mixture-of-experts with top-k gating, which the FO-MoE adapts for fairness."},{"cited_title":"An image is worth 16x16 words: Trans- formers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Defines the vision transformer structure whose last attention block and class token FO-MoE modifies."},{"cited_title":"Jacobs, Michael I","cited_arxiv_id":null,"evidence_quote":"Introduces the mixture-of-experts gating formulation that the paper's expert weighting scheme builds on."}],"review_version":1}