{"id":"270f4064-adae-4610-a8c3-71b6919bab8a","arxiv_id":"2411.11939","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FairDi trains biased per-group teachers and distills them into one student, reporting better accuracy-fairness trade-offs than prior methods on medical imaging benchmarks.","lead":"This paper proposes FairDi, a training method that uses group-specific teacher models to distill fair knowledge into one student model for medical image classification and segmentation. It reports higher overall and group-specific accuracy and smaller performance gaps than existing fairness methods on five medical imaging datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported superiority lacks repeated-run variance and contains inconsistent baseline numbers; the central empirical claim is not yet supported.","rationale":"The reader's weakest_assumption concerns the capacity of a single shared linear classification layer on the frozen FIS backbone to approximate per-group teacher outputs. That is a genuine theoretical gap, but it is not the most load-bearing risk because the paper's empirical results, if reliable, would already demonstrate that the shared head is sufficient in practice. The stronger risk is that the measurements themselves are not yet reliable enough to support the central superiority claim. The paper reports no repeated runs or error bars, and its own ablation and main tables contain inconsistent ERM values for the same dataset and attribute. The near-zero AUC Gap of 0.000029 on HAM10000 Gender is below what a single test set of roughly 1,000 images can resolve, so it cannot be interpreted without confidence intervals. The comparison may also be unfair if CutMix was applied only to FairDi and not to all baselines, given that the ablation shows CutMix alone can move Overall AUC by several points. FairDi also loses on some fairness cells, so the 'outperforms all' claim depends on aggregated averages. These issues are already reflected in the reader's conditional verdict and its rationale, so I do not propose moving the verdict: the paper needs code, seeds, error bars, and a clarified baseline protocol before the central claim can be accepted. I partially agree with the reader's identified weakest assumption, but I regard the empirical reliability of the comparison as the more load-bearing concern.","tokens_in":22766,"tokens_out":8337,"duration_ms":83024,"concrete_test":"Run all methods (ERM, GroupDRO, SWAD, FIS, FairDi) on HAM10000 Gender and PAPILA Gender with the same five train/val/test splits and the same CutMix usage for every method; report mean and standard deviation over five seeds for Overall AUC, Minimum AUC, and AUC Gap. If FairDi's margin over FIS is within one standard deviation on either dataset, the headline claim of simultaneous improvement is not established; if it exceeds two standard deviations, the concern is resolved for those benchmarks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is empirical: FairDi improves overall accuracy, minimum group accuracy, and fairness over FIS and other methods (Table 1). For that claim to hold, the benchmark comparison must be reliable, and it is not, for three concrete reasons. First, no seeds or error bars are reported; all numbers are single runs. Table 3 reports ERM HAM10000 Gender Overall AUC 0.8890 and PAPILA Gender 0.8433, while Table 11 lists 0.8520 and 0.7840 for the same method and attribute, implying uncontrolled splits or runs. Second, the ablation's HAM10000 Gender AUC Gap of 0.000029 (Table 3) is far below the resolution of a single test set of about 1,000 images; without variance estimates, this value cannot support a fairness claim. Third, the paper states that CutMix is used for backbone, teacher, and student models, but it does not state whether every baseline uses CutMix; the ablation shows CutMix alone changes ERM PAPILA Gender Overall AUC from 0.8433 to 0.9142. Additionally, FairDi does not win every reported cell: FIS has a lower AUC Gap on MIMIC-CXR Race (0.0103 vs 0.0160) and CheXpert Age (0.0256 vs 0.0281), so the 'outperforms all' claim rests on averages over mostly favorable rows. A Nemenyi test on one run per dataset cannot establish significance. If repeated runs show overlapping confidence intervals, the central claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Fair Distillation (FairDi), a two-stage knowledge-distillation method for fairness in medical imaging classification and segmentation. Step 0 trains a backbone with the FIS loss; Step 1 fine-tunes one linear classification head per demographic group on the frozen backbone to create biased group-specific teachers; Step 2 trains a single student head with a KL distillation loss to the per-group teachers combined with the FIS loss. The authors claim that FairDi is the first fairness method to decompose group-accuracy maximization from overall-accuracy and inter-group-gap optimization, and report improved overall AUC, minimum-group AUC, ES-AUC, and reduced AUC Gap relative to ERM, GroupDRO, SWAD, and FIS on five classification datasets, plus improved Dice/IoU and equity-scaled metrics on the Harvard-FairSeg segmentation dataset.","tokens_in":23063,"tokens_out":4537,"duration_ms":44592,"significance":"The proposed decomposition is conceptually clean and practically attractive: if the reported gains hold, FairDi would offer a simple and adaptable way to improve both accuracy and fairness by using biased teachers, and its application to both classification and segmentation gives it broad potential impact. The paper also has strengths: the method is clearly specified, the ablation study isolates the contributions of CutMix, FIS, and the teacher-student steps, and the authors include detailed supplementary tables and fairness metric definitions. However, the central empirical claim currently rests on single-split, single-run results with no variance information, and there are internal inconsistencies between the main tables and the supplementary table. These problems are load-bearing because the paper's contribution is primarily empirical superiority over prior methods.","major_comments":[{"comment":"The same baseline is reported with different numbers across tables: ERM HAM10000 Gender Overall AUC is 0.8890 in Table 3 but 0.8520 in Table 11, and PAPILA Gender ERM Overall AUC is 0.8433 in Table 3 but 0.7840 in Table 11. The manuscript does not explain whether these are different splits, different seeds, or different preprocessing choices. Without a clear protocol, the single-split averages in Table 1 are not reproducible and the cross-method comparison is undermined.","section":"§4.4, Table 3 vs. §9, Table 11"},{"comment":"All reported classification results appear to be single runs on a single split; no seeds, confidence intervals, or error bars are reported. The Friedman/Nemenyi tests in Section 4.2 and Figure 3 use one observation per algorithm per dataset, so the statement that FairDi is 'significantly better than all others' for worst-case AUC is not statistically supported. Please provide repeated-seed results with variance estimates and base significance claims on distributions over runs, or state explicitly that the CD diagrams are illustrative only.","section":"§4.2 and Figure 3"},{"comment":"The paper states that CutMix is used for the backbone, teacher, and student models, but it does not state whether the ERM, GroupDRO, SWAD, and FIS baselines also use CutMix. The ablation in Section 4.4 shows that adding CutMix changes ERM PAPILA Gender Overall AUC from 0.8433 to 0.9142 and AUC Gap from 0.0180 to 0.1500; if the baselines were trained without CutMix, a substantial part of the reported improvement could be due to augmentation rather than the FairDi mechanism. Please specify the exact training protocol for every baseline.","section":"§4.1 and Table 3"},{"comment":"Table 1 lists identical average rank values for AUC Gap, MeanPSD, and MaxPSD for every method (3.00, 3.55, 3.27, 3.00, 2.18), even though the underlying per-dataset values in Table 11 are clearly different and would not be expected to yield identical ranks across three distinct fairness metrics. This strongly suggests a calculation or transcription error in the rank rows; please recompute and verify all rank statistics.","section":"Table 1"},{"comment":"The claim that FairDi 'outperforms existing methods' is stronger than the evidence in Table 11: FIS has a lower AUC Gap than FairDi on MIMIC-CXR Race (0.0103 vs. 0.0160) and CheXpert Age (0.0256 vs. 0.0281), and FairDi does not win every reported cell. The averages in Table 1 hide these exceptions. Please qualify the claim to 'highest average' and provide a per-cell comparison or significance test before asserting global superiority.","section":"§4.2, Table 11"},{"comment":"Equation (3) trains a single linear classification head on the frozen FIS backbone to match the per-group teacher linear heads via KL distillation, yet the manuscript provides no capacity or geometry argument that a single linear map can realize the group-specific teacher functions on the shared features. If the features require different linear maps per group, the per-sample KL term averages conflicting teacher targets and could degrade both accuracy and fairness. Please provide an analysis of this assumption or an experiment varying the student head capacity.","section":"§3.3, Eq. (3)"}],"minor_comments":[{"comment":"Abstract contains 'F air Distillation' with an erroneous space; it should be 'Fair Distillation'.","section":"Abstract"},{"comment":"The KL divergence is written as KL[h_student || h_teacher], but the arguments are not explicitly defined as probability distributions over classes; please clarify that h outputs are softmax probabilities and state how the temperature τ enters the KL term.","section":"§3.3, Eq. (3)"},{"comment":"The dataset name is spelled 'Papila' in the datasets list but 'PAPILA' elsewhere; please use consistent spelling.","section":"§4.1, Datasets"},{"comment":"The caption says 'Each model's Pareto front includes two points: one maximizing worst-group AUC, the other maximizing overall AUC,' which is unclear because a Pareto front is a set of solutions, not two points; please clarify how the two points are selected and what they represent.","section":"Figure 1 caption"},{"comment":"The ES-AUC definition uses AUC(D_g) but D_g is not defined in the supplementary; please add the definition or reference the main text definition from Section 3.2.","section":"Supplementary, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is worth pursuing, but the empirical validation needs substantial work before the claims can be accepted. The identical rank rows for AUC Gap, MeanPSD, and MaxPSD in Table 1 are a particular red flag and should be checked carefully; if those numbers cannot be reproduced, the central empirical claim would be in jeopardy. Also, the method builds on FIS [31], which shares an author with this submission; the comparison against FIS should be independently calibrated and reported in full."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FairDi is a simple, sensible idea: pre-train a backbone with FIS, add one linear teacher head per demographic group on the frozen backbone, then train a single student head by distilling per-sample from the matching teacher plus FIS loss. The specific combination is new in medical imaging, and the paper's framing—decoupling group-specific accuracy from overall accuracy and gap minimization—is fair. What earns credit is scope: five classification datasets, multiple sensitive attributes, a segmentation dataset, full supplementary tables, and ablations that roughly make sense. The method does not look circular; teachers are trained on group subsets and the student is evaluated on held-out data.\n\nThe problem is that the central empirical claim rests on numbers that don't hold together. Table 3 lists ERM HAM10000 Gender Overall AUC as 0.8890; Table 11 lists 0.8520 for the same cell. PAPILA Gender ERM is 0.8433 in the ablation and 0.7840 in the full table. That is not a rounding difference. Either the splits or the runs are different and the paper doesn't say so. Second, no seeds or error bars are reported anywhere; the Nemenyi CD diagrams on one run per dataset can't support 'significantly better' claims. Third, the ablation shows CutMix alone moves PAPILA ERM Overall AUC from 0.8433 to 0.9142, and the paper never states whether every baseline used CutMix. Until that is pinned down, part of the reported margin is unexplained. Finally, the HAM10000 Gender AUC Gap of 0.000029 is implausibly small for a test set of roughly a thousand images; with no variance estimate, it shouldn't be presented as a meaningful fairness result. The heavy reliance on FIS, co-authored by a current author, is not itself a flaw, but the overlap should be stated explicitly.\n\nNone of this sinks the method. The idea is plausible and the consistent direction of the averages across many settings suggests something real is going on. But 'outperforms all methods' is not supported. The paper needs reruns, variance, a clear statement of exactly which augmentation each baseline received, and a reconciliation of the inconsistent tables.\n\nThis is a paper for a serious referee, not a desk reject. I'd send it to review with a request for those corrections. I wouldn't yet cite the quantitative results in my own work.","headline":"FairDi is a plausible distillation-based fairness method whose broad experiments are undermined by missing variance and inconsistent baseline numbers; the idea deserves review but the central claim is not yet supported.","tokens_in":23642,"tokens_out":2578,"would_cite":false,"duration_ms":24945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FairDi, a teacher-student fairness method, claims to lift overall accuracy and shrink group gaps at the same time in medical imaging.","keywords":["FairDi","fairness in medical imaging","knowledge distillation","group fairness","AUC gap","teacher-student learning","medical image classification","medical image segmentation"],"falsifier":"Train FairDi on a synthetic two-group classification problem in which the optimal group-specific decision boundaries on the frozen feature space provably cross (for example, mirrored class-conditional Gaussians per group). If the student's group-specific AUCs fall substantially below the teachers' group-specific AUCs while the AUC gap remains above the teacher gap, the single-shared-head assumption is refuted; a null result would support the paper's claim that the shared head can carry all group-specific functions.","tokens_in":22536,"feed_emoji":"⚖️","tokens_out":7122,"duration_ms":64757,"temperature":0.7,"pith_summary":"The paper introduces FairDi, a fairness method for medical image classification and segmentation that separates three goals that usually fight each other: overall accuracy, accuracy for each demographic group, and small accuracy differences between groups. FairDi first trains a shared backbone with a fairness-aware loss, then fine-tunes a separate 'teacher' classification head for each demographic group to maximize that group's accuracy, and finally trains one 'student' head that distills the teachers' softened predictions while also being trained for overall accuracy and fairness. On five chest, skin, and eye classification benchmarks and one segmentation benchmark, the paper reports that FairDi achieves the best average overall, worst-group, and equity-scaled AUC, and the smallest average AUC gap, among the compared methods. The significance, if the results hold, is that fairness does not have to be bought by sacrificing accuracy; the standard trade-off can be shifted by decoupling group-optimized experts from a unified model.","feed_headline":"Distilling biased teachers lifts medical AI accuracy and fairness","feed_subtitle":"Teacher-student FairDi posts top overall AUC and smallest group gap across five medical datasets.","key_machinery":"The machinery is a three-step pipeline built on a frozen feature extractor $f_{\\theta^*}$ trained with the Fair Identity Scaling (FIS) loss, a loss that reweights each sample by individual difficulty and by the optimal-transport distance between a group's loss distribution and the overall loss distribution. Step 1 adds a randomly initialized classification head $h_{\\phi_g^{(t)}}$ for each demographic group $g$ and fine-tunes only that head on the group's samples to maximize group-specific accuracy. Step 2 adds a student head $h_{\\phi^{(s)}}$ and minimizes $\\lambda\\tau^2\\,\\mathrm{KL}\\big(h_{\\phi^{(s)}}(f_{\\theta^*}(x))\\,\\|\\,h_{\\phi_a^{(t)},\\tau}(f_{\\theta^*}(x))\\big) + (1-\\lambda)\\,w(x,y,a,c)\\,\\ell\\big(h_{\\phi^{(s)}}(f_{\\theta^*}(x)), y\\big)$ with $\\lambda=0.95$ and $\\tau=1.5$. The per-sample selection of teacher $a$ is what lets the student inherit each group's knowledge; the FIS term is what keeps the student from drifting toward any single teacher and anchors overall accuracy and fairness.","core_discovery":"The central claim is that a teacher-student decomposition changes the fairness-accuracy Pareto front in medical imaging. Instead of one model trying to satisfy all objectives at once, FairDi trains a teacher head per demographic cohort on a frozen backbone, each greedily optimized only for its own group's classification accuracy, then trains a single student head that, for every sample, matches the softened output of that sample's group teacher through a KL divergence term while also minimizing a sample-weighted classification loss. The paper reports that this yields the best average overall AUC (0.9137), best average minimum subgroup AUC (0.9050), best average equity-scaled AUC (0.9062), and smallest average AUC gap (0.0166) across HAM10000, Fitzpatrick17k, PAPILA, CheXpert, and MIMIC-CXR, and similarly improved equity-scaled Dice for optic-cup and optic-rim segmentation, relative to the compared fairness methods. The claim is that FairDi is the first fairness method to decompose group-accuracy maximization from overall-accuracy maximization and inter-group gap minimization.","pith_inferences":["The paper does not test intersectional groups, but nothing in the method forbids defining a teacher for each intersection of sensitive attributes; however, the single-shared-head assumption becomes harder to satisfy as the number of cohorts grows, so the method's advantage may shrink with more groups.","A direct probe of the capacity assumption would be to report student versus teacher per-group AUCs on the same held-out data; if the student closely tracks its teachers, the distillation is faithfully transferring knowledge, and if not, the gap between them quantifies the cost of the shared head.","The frozen-backbone design suggests FairDi could be combined with any representation learning scheme, not just the FIS-trained backbone, although the fairness of the final student would then depend on whether the frozen features preserve enough group-specific information for the teachers to excel."],"forward_implications":["FairDi reports the best average overall AUC (0.9137) and best average minimum-group AUC (0.9050) across the five classification benchmarks, so a practitioner wanting both accuracy and worst-group protection would get both from one training scheme.","The average AUC gap drops to 0.0166, below the best previously reported gap, meaning the margin between the strongest and weakest demographic cohort nearly disappears on average.","On the FairSeg segmentation benchmark, FairDi reports the best average overall Dice (0.8260) and equity-scaled Dice (0.8062), so the decomposition transfers from binary classification to dense pixel prediction.","Since only the small classification heads are trainable in Steps 1 and 2, FairDi can be applied on top of an existing frozen medical imaging backbone with total training time around 40 minutes on HAM10000 and no added test-time cost."],"supporting_citations":[{"why":"Supplies the Fair Identity Scaling loss used for the Step-0 backbone and serves as the strongest classification baseline.","marker":"[31]"},{"why":"Defines the benchmark datasets, preprocessing, and evaluation metrics (AUC gap, worst-case AUC, statistical tests) used in the comparisons.","marker":"[64]"},{"why":"Provides the softened-logit knowledge-distillation formulation that the student's KL term implements.","marker":"[15]"},{"why":"GroupDRO is a worst-group baseline that FairDi must beat on classification.","marker":"[46]"},{"why":"SWAD is a flat-minima domain-generalization baseline for classification fairness.","marker":"[2]"},{"why":"Provides the FairSeg dataset and the FEBS fairness baseline used for the segmentation experiments.","marker":"[54]"},{"why":"TransUNet is the segmentation backbone that FairDi is built on and compared with.","marker":"[4]"},{"why":"ERM is the standard empirical-risk-minimization baseline for classification.","marker":"[57]"}],"fun_headline_variants":["FairDi distills biased teachers to boost medical AI fairness","Teacher-student trick improves medical AI fairness and accuracy","FairDi tops fairness and accuracy in medical imaging","Biased teachers, fair student: new method for medical AI","FairDi: distill group experts for fairer medical AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The student is a single linear classification head on the same frozen features as the teachers, so the method stands or falls on whether one shared head can approximate every group-specific teacher's outputs for every sample; if the features force the per-group classifiers to disagree too strongly, the distillation term averages conflicting targets and the student loses both accuracy and fairness.","fun_headline_variants_meta":{"raw":{"variants":["FairDi distills biased teachers to boost medical AI fairness","Teacher-student trick improves medical AI fairness and accuracy","FairDi tops fairness and accuracy in medical imaging","Biased teachers, fair student: new method for medical AI","FairDi: distill group experts for fairer medical AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1327,"prompt_tokens":959,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":575,"tokens_out":368,"duration_ms":4279,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:12:49.740353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train FairDi on a synthetic two-group classification problem in which the optimal group-specific decision boundaries on the frozen feature space provably cross (for example, mirrored class-conditional Gaussians per group). If the student's group-specific AUCs fall substantially below the teachers' group-specific AUCs while the AUC gap remains above the teacher gap, the single-shared-head assumption is refuted; a null result would support the paper's claim that the shared head can carry all group-specific functions.","supporting_citations":[{"cited_title":"Fairvi- sion: Equitable deep learning for eye disease screening via fair identity scaling, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Fair Identity Scaling loss used for the Step-0 backbone and serves as the strongest classification baseline."},{"cited_title":"Swad: Domain generalization by seeking flat minima","cited_arxiv_id":null,"evidence_quote":"SWAD is a flat-minima domain-generalization baseline for classification fairness."},{"cited_title":"Fairseg: A large-scale medical image seg- mentation dataset for fairness learning using segment any- thing model with fair error-bound scaling","cited_arxiv_id":null,"evidence_quote":"Provides the FairSeg dataset and the FEBS fairness baseline used for the segmentation experiments."}],"review_version":1}