{"id":"0f36eb4e-5e16-4af9-b5f6-3421a5807e5e","arxiv_id":"2502.01188","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"FairUDT uses uplift-style divergence splitting and selective leaf relabeling to reduce demographic parity and average odds gaps on three fairness benchmarks, with test-set relabeling in its main evaluation.","lead":"This paper presents FairUDT, decision trees that split by how much favored and deprived groups differ in their outcomes, then relabel some records in high-difference leaves to reduce measured discrimination. It reports fairness gains on Adult, COMPAS, and German Credit, but the main evaluation changes test labels, so the headline results are not trustworthy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline AOD=0 is an artifact of relabeling the test set: Table 3's 'Relabelled Test Set' rows measure fairness against labels the method itself has altered, so the claimed state-of-the-art tradeoff is not supported by comparable evidence.","rationale":"The reader's verdict is REJECT, and I agree with that verdict, though via a slightly different emphasis. The reader's 'weakest_assumption' names the causal/confounding assumption behind equating class-probability gaps with discrimination; that is a real limitation for the discrimination-identification framing. However, the empirical claim that would make the paper publishable—the acceptable accuracy–discrimination tradeoff and the AOD=0 results—fails earlier and more directly because the fairness metrics in Table 3 are computed on test labels that the method relabels. Fairness evaluation on modified labels is not a measurement of the original data's discrimination; it is consistency with the modified labels. This issue is decisive regardless of how one resolves the causal question. Therefore the minimal check is to rerun with original test labels only. The reader's rationale mentions the relabeled-test-set problem, so agreement is partial rather than full; the weakest-assumption field points to the causal issue instead. I would keep the REJECT verdict: the central claim is not supported by the evidence as reported, although the raw-test-set results suggest the underlying tree-and-relabeling idea may have something worth further study under a corrected protocol.","tokens_in":20859,"tokens_out":6531,"duration_ms":66549,"concrete_test":"Run the released FairUDT code on the same Adult/COMPAS/German splits, train LR as in Section 5, and evaluate once on the original test labels and once on the paper's relabeled test labels. Recompute DP and AOD for both sets of labels. If the AOD drops from 0.04/0.02/0.02 (raw) to 0.00/-0.01 (relabeled), the claimed AOD=0 is a label-construction artifact; additionally compare those raw-label results against the AIF360 baselines on original labels to see whether the state-of-the-art claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central support for the paper's headline claims is Table 3's 'FairUDT + Relabelled Test Set' rows. Section 4 says the test data 'must also be pre-processed before making predictions with the classifier,' and the relabeling procedure is designed to impose P_F(y+|l)=P_D(y+|l) and P_F(y-|l)=P_D(y-|l) in each discriminatory leaf. Evaluating AOD against these modified test labels measures how well the classifier fits labels that the method has already pushed toward group parity, not how much discrimination remains in the original data. The Adult row's AOD 0.00, and the COMPAS/German relabeled rows, are therefore not comparable to the AOD values reported for Raw, Disparate Impact Remover, Optimized Pre-processing, or Reweighing, all of which use original labels. The legitimate 'Raw Test Set' numbers are weaker (Adult AOD 0.04, COMPAS 0.02, German 0.02) and do not by themselves establish state-of-the-art DP/AOD. A second inconsistency compounds this: Algorithm 1's promotion/demotion count p (lines 13 and 20) mixes a conditional probability with what should be a count, so even the intended equality conditions are not correctly implemented as printed. The evaluation artifact alone is sufficient to invalidate the paper's main empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FairUDT, a decision-tree-based data pre-processor that uses divergence between favored and deprived class distributions to grow a tree, identifies leaves with a positive discrimination score discl, and then relabels selected records so that favored and deprived positive/negative rates match in those leaves. The pre-processed data are used to train classifiers, and the method is evaluated on Adult, COMPAS, and German Credit against pre-processing baselines using DP, AOD, BA, and accuracy. The paper claims state-of-the-art fairness results, including AOD 0.00 on the relabeled Adult test set, as well as better interpretability than standard decision trees.","tokens_in":21183,"tokens_out":6979,"duration_ms":75806,"significance":"The idea of adapting uplift modeling to discrimination identification is potentially interesting, and the paper provides a concrete tree-splitting framework with KL and Euclidean divergence gains, a proposed leaf relabeling procedure, and a human-subject interpretability comparison. The code is made available, which is a strength. However, the empirical claims do not currently survive scrutiny: the headline fairness numbers are computed on test labels that the method itself has altered, the relabeling algorithm as printed contains a dimensional error in its count formulas, and the favored/deprived comparison is treated as causal discrimination without supporting assumptions. These issues are load-bearing for the paper's central claim of an acceptable, state-of-the-art accuracy-discrimination tradeoff, so the current evidence does not justify acceptance.","major_comments":[{"comment":"The 'FairUDT + Relabelled Test Set' rows in Table 3 are not a valid evaluation of discrimination removal. Section 4 states that 'the test data must also be pre-processed before making predictions with the classifier,' and Algorithm 1 is designed to enforce P_F(y+|l)=P_D(y+|l) and P_F(y-|l)=P_D(y-|l) in selected leaves. AOD computed on these modified test labels therefore measures how well the classifier predicts labels that the method has already pushed toward group parity, not how much discrimination remains in the original data. The reported AOD of 0.00 for Adult is an artifact of this procedure, and it is not comparable to the AOD values reported for Disparate Impact Remover, Optimized Pre-processing, or Reweighing, which are evaluated on original test labels. The only comparable evidence is the 'FairUDT + Raw Test Set' rows (Adult AOD 0.04, COMPAS 0.02, German 0.02), and those numbers do not by themselves establish the claimed state-of-the-art tradeoff.","section":"§4, §6.1, Table 3"},{"comment":"The formulas for the number of records to relabel mix probabilities with counts. In Promote, p is computed as floor(P(y+|sF)P(sD) - P(y+|sD)), and in Demote as floor(P(y-|sD)P(sF) - P(y-|sF)). These expressions have probability units, not count units, and they do not include the leaf-specific number of deprived or favored records. To impose P_F(y+|l)=P_D(y+|l), the number of deprived negative records to promote should be N_D^l(P_F(y+|l)-P_D(y+|l)) (and analogously for demotion), with an additional check that this number does not exceed the available records in the leaf. As printed, the algorithm cannot be implemented as intended and is not reproducible.","section":"Algorithm 1, lines 13 and 20"},{"comment":"The paper defines discrimination as the class-probability difference between favored and deprived groups sharing the same characteristics, and then uses that criterion to relabel data. This is a valid statistical parity criterion only if the favored/deprired split is treated as a randomized or conditionally ignorable treatment; the paper does not state or test this causal assumption, nor does it control for unmeasured confounders. As written, the method can remove any disparity correlated with the sensitive attribute, including disparities that reflect legitimate qualification differences. The central claim that FairUDT removes discrimination rather than group disparity therefore needs either an explicit causal identification argument or a narrower, clearly stated statistical-disparity interpretation.","section":"§2, §3.4, Eq. (3.11)"},{"comment":"The discrimination threshold sigma_t is tuned per dataset to the reported optimum (Adult 0.61, COMPAS 0.1, German 1.64), and the paper states that these are the values used for the final results. Because the same test sets are used for tuning and for reporting, the best DP/AOD numbers in Table 3 are selected values rather than predictive estimates. Table 3 also reports no confidence intervals or significance tests. To establish the claimed tradeoff, the authors should report results on a validation split, or show the full sigma_t sweep with confidence bands and a clear separation between tuning and evaluation.","section":"§6.1, Figure 3"}],"minor_comments":[{"comment":"The dataset name 'COMP AS' appears to be a typo for 'COMPAS'.","section":"Table 2"},{"comment":"The sentence replacing dy(a) with 'dy (class probabilities test independence for favored distribution)' should refer to the deprived distribution, not the favored one.","section":"Appendix B, Proposition 3.2 proof"},{"comment":"Definition 2 says a leaf is relabeled only if discl > sigma_t, while Algorithm 1 uses disc(l) >= sigma_t; the inconsistency should be resolved, especially for sigma_t = 0.","section":"§3.4, Algorithm 1 line 6"},{"comment":"The relationship between the 75-25 train-test split and the 10-fold cross-validation is not clear; the paper should state whether cross-validation is applied within the training portion and how the test set is used across folds.","section":"§5.2"},{"comment":"The text says AUC is lower after pre-processing, but Figure 2 only shows ROC curves and no AUC values are reported; the numerical AUC values should be given.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"The test-set relabeling issue is the core problem: the headline AOD=0.00 is produced by the evaluation protocol, not by the model's independent performance. Even a thorough revision would need to re-run all experiments on original test labels and re-derive the relabeling counts before the central empirical claim could be assessed. The paper may contain useful ideas, but the current manuscript does not support the claimed results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is genuine: using uplift-style divergence splitting to find discriminatory leaves and then selectively relabeling is a reasonable new combination, and the paper adapts the Rzepakowski–Jaroszewicz criteria with sensible normalization for imbalanced favored/deprived groups. The critique of Kamiran et al.'s old discrimination measure is fair. Second, the headline AOD=0.00 is an artifact of relabeling the test set. Section 4 explicitly says the test data must be pre-processed, and the 'Relabelled Test Set' rows in Table 3 evaluate against labels the method itself has altered. That is not comparable to the baselines, which use original labels. The raw test set numbers—Adult AOD 0.04, COMPAS 0.02, German 0.02—are decent but not state-of-the-art, and the claim of 'state-of-the-art results for both DP and AOD on COMPAS and German' only holds on the relabeled rows.\n\nThere is also a concrete bug in Algorithm 1: the number of records to relabel, p, is computed as a floor of a probability expression, not a count. Lines 13 and 20 mix conditional probabilities with what should be record counts, so the stated equality conditions are not correctly implemented as printed.\n\nSigma_t is chosen per dataset to hit the reported optimum, which is test-set tuning. The paper openly says 'all the datasets are pre-processed using the optimal discrimination thresholds,' and Figure 3 shows the thresholds were selected after seeing the fairness metrics. That undercuts the generality claim.\n\nThe paper equates class-probability differences with discrimination without testing the causal assumption that favored/deprived status is as-if random. That is common in fairness pre-processing, so I treat it as a minor concern, but it means the method can remove legitimate signal rather than bias.\n\nThe interpretability section is honest but the trees are not that sparse: 3,813 nodes on Adult is a lot, and the simulatability rating is based on four graduates rating 30 subgroups.\n\nWho is this for? Someone working on fairness pre-processing and interpretable trees could mine the splitting criteria, but they should not cite the headline results. The paper deserved a serious referee—the idea is substantive—but the evaluation needs to be redone on original test labels, Algorithm 1 fixed, and sigma_t selected on a validation set. If you are looking for a worked example of test-set contamination in fairness evaluation, this is a useful object lesson.","headline":"Real idea, invalid headline result: the AOD=0.00 is an artifact of relabeling the test set, while the raw-test-set numbers are decent but not state-of-the-art.","tokens_in":21675,"tokens_out":2551,"would_cite":false,"duration_ms":26772,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"FairUDT claims that a decision tree grown to maximize divergence between favored and deprived groups' class probabilities can pinpoint discriminatory subgroups, and that relabeling only those leaves removes bias while keeping accuracy.","keywords":["algorithmic fairness","discrimination identification","uplift modeling","decision trees","leaf relabeling","demographic parity","average odds difference","interpretable AI"],"falsifier":"Run FairUDT on synthetic data with equal ground-truth outcomes across groups but a strong confounder correlated with the sensitive attribute; if the tree flags leaves and relabels them, the identification step has detected association, not discrimination, falsifying the central claim for that setting.","tokens_in":20713,"feed_emoji":"⚖️","tokens_out":6883,"duration_ms":65904,"temperature":0.7,"pith_summary":"FairUDT claims that bias can be located by treating the favored group as a treatment arm and the deprived group as a control arm, then growing a decision tree whose splits maximize the divergence between the two groups' class-probability distributions. The leaves that end up with a large favored-minus-deprived probability gap are flagged as discriminatory regions, and a relabeling step flips only records in those leaves until the gap closes. Because relabeling is selective rather than whole-group, the paper argues, fairness improves without the usual collapse in accuracy. On Adult, COMPAS, and German Credit used as preprocessed inputs to standard classifiers, the method reports a perfect average odds difference on the relabeled Adult test set and state-of-the-art demographic parity and average odds difference on COMPAS and German Credit. The same trees double as an interpretable audit tool: the leaf paths spell out which attribute combinations form the biased subgroups.","feed_headline":"Decision tree finds and fixes bias subgroup by subgroup","feed_subtitle":"Divergence-splitting tree relabels only biased leaves, keeping accuracy while cutting discrimination.","key_machinery":"The engine is the divergence gain $D_{\\mathrm{gain}}(A) = D(P^F(Y):P^D(Y)|A) - D(P^F(Y):P^D(Y))$, where $D$ is either KL divergence or squared Euclidean distance, normalized by a split-information term that penalizes splits separating the favored and deprived groups into different subtrees and splits with many outcomes. KL divergence is directional, so it reports how much the favored distribution diverges from the deprived reference distribution, which the paper argues is exactly the direction of preferential treatment. The companion mechanism is leaf relabeling: only leaves with $discl > \\sigma_t$ are edited, and within those leaves only enough randomly chosen records are promoted or demoted to force $P^F(y+|l)=P^D(y+|l)$ and $P^F(y-|l)=P^D(y-|l)$; the threshold $\\sigma_t \\in [0,2]$ is the accuracy-fairness dial.","core_discovery":"The paper's central claim is that uplift modeling can be repurposed from marketing to discrimination identification: when the favored group is treated as treatment and the deprived group as control, the divergence $D(P^F(Y):P^D(Y)|A)$ before and after a split measures how much a feature separates the groups' outcomes, and maximizing that divergence grows a tree whose leaves are precisely the subgroups where favoritism concentrates. At each leaf the paper defines discrimination as $discl = (P^F(y+|l)-P^D(y+|l)) + (P^D(y-|l)-P^F(y-|l))$, relabeling a leaf only when this exceeds a tunable threshold $\\sigma_t$, by promoting deprived negative instances when the leaf's majority class is positive and demoting favored positive instances when it is negative. The paper reports that this procedure, run with KL-divergence splitting and logistic regression, reaches AOD 0.00 on the relabeled Adult test set and state-of-the-art DP and AOD on COMPAS and German Credit, with accuracy equal to or better than raw data in those cases. It also claims the resulting trees are sparser and shallower than ordinary decision trees, making the detected discriminatory subgroups directly readable as attribute rules.","pith_inferences":["If the favored/deprived gap is not caused by preferential treatment but by an unmeasured confounder or by genuine differences in qualifications, the leaf relabeling will edit labels that are not biased; a causal or counterfactual check on the identified leaves would be needed before the method's 'discrimination' claim is used for real decisions.","A synthetic-data test with known ground-truth bias could separate detection from confounding: inject a gap only through a confounder and see whether FairUDT relabels; if it does, the divergence criterion is measuring association, not discrimination.","The same divergence machinery could be extended to multi-valued sensitive attributes by replacing the binary favored/deprived split with multiple treatment arms, which the paper lists as future work but does not develop."],"forward_implications":["Used as a pre-processor, FairUDT lets any downstream classifier inherit lower group disparity without changing the classifier, since the relabeled dataset is what is trained on.","On COMPAS and German Credit, FairUDT beats the compared pre-processing baselines on DP and AOD while keeping balanced accuracy at or near the raw-data level, so the tradeoff is acceptable rather than prohibitive.","Because relabeling targets only leaves above the threshold, it preserves more of the original labels than whole-group relabeling, which is why the paper reports raw-level accuracy on Adult and a 1% accuracy gain on German Credit.","The interpretability result means the method can be used as an audit tool: each discriminatory leaf corresponds to an if-then subgroup that a human can read and verify.","The tunable threshold $\\sigma_t$ gives practitioners a single knob to dial between fairness and accuracy at subgroup level rather than across the whole dataset."],"supporting_citations":[{"why":"Supplies the uplift-modeling divergence gains (KL and squared Euclidean) that FairUDT adapts to favored-versus-deprived group probabilities.","marker":"[41]"},{"why":"Defines the discrimination-aware decision-tree splitting and leaf relabeling baseline that FairUDT extends and compares against.","marker":"[9]"},{"why":"Adult is one of the three benchmark datasets used to evaluate the preprocessing pipeline.","marker":"[16]"},{"why":"COMPAS is the recidivism dataset on which FairUDT reports zero demographic parity at $\\sigma_t=0.1$.","marker":"[17]"},{"why":"German Credit is the third benchmark dataset, on which FairUDT holds balanced accuracy while lowering discrimination.","marker":"[18]"},{"why":"Optimized pre-processing is a comparison baseline FairUDT must match or beat on the accuracy-fairness tradeoff.","marker":"[5]"},{"why":"Disparate impact remover is a comparison baseline for the same tradeoff on all three datasets.","marker":"[6]"},{"why":"Reweighing is the third preprocessing baseline FairUDT is compared with in Table 3.","marker":"[20]"},{"why":"Causal-tree work for discovering discriminatory subgroups, which FairUDT contrasts with as the closest existing use of causal structure.","marker":"[42]"},{"why":"Shows recursive partitioning for heterogeneous treatment effects needs balanced treatment and control counts, the constraint FairUDT avoids by using probability differences.","marker":"[43]"}],"fun_headline_variants":["Divergence-splitting tree relabels only biased leaves","Uplift tree pinpoints bias, then corrects it","Fair tree splits on divergence to fix favoritism","Tree uses uplift to locate and mend biased groups","Uplift-based tree finds and fixes unfair subgroups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a class-probability gap between favored and deprived groups at a leaf is discrimination; if that gap comes from confounding or real group differences rather than unfair treatment, relabeling removes signal instead of bias.","fun_headline_variants_meta":{"raw":{"variants":["Divergence-splitting tree relabels only biased leaves","Uplift tree pinpoints bias, then corrects it","Fair tree splits on divergence to fix favoritism","Tree uses uplift to locate and mend biased groups","Uplift-based tree finds and fixes unfair subgroups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2514,"prompt_tokens":981,"completion_tokens":1533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1454}},"tokens_in":597,"tokens_out":1533,"duration_ms":13343,"temperature":1.0,"reasoning_tokens":1454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:15:57.437357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FairUDT on synthetic data with equal ground-truth outcomes across groups but a strong confounder correlated with the sensitive attribute; if the tree flags leaves and relabels them, the identification step has detected association, not discrimination, falsifying the central claim for that setting.","supporting_citations":[{"cited_title":"Rzepakowski, S","cited_arxiv_id":null,"evidence_quote":"Supplies the uplift-modeling divergence gains (KL and squared Euclidean) that FairUDT adapts to favored-versus-deprived group probabilities."},{"cited_title":"Kamiran, T","cited_arxiv_id":null,"evidence_quote":"Defines the discrimination-aware decision-tree splitting and leaf relabeling baseline that FairUDT extends and compares against."},{"cited_title":"Larson, M","cited_arxiv_id":null,"evidence_quote":"COMPAS is the recidivism dataset on which FairUDT reports zero demographic parity at $\\sigma_t=0.1$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"German Credit is the third benchmark dataset, on which FairUDT holds balanced accuracy while lowering discrimination."},{"cited_title":"Calmon, D","cited_arxiv_id":null,"evidence_quote":"Optimized pre-processing is a comparison baseline FairUDT must match or beat on the accuracy-fairness tradeoff."},{"cited_title":"Feldman, S","cited_arxiv_id":null,"evidence_quote":"Disparate impact remover is a comparison baseline for the same tradeoff on all three datasets."},{"cited_title":"Kamiran, T","cited_arxiv_id":null,"evidence_quote":"Reweighing is the third preprocessing baseline FairUDT is compared with in Table 3."},{"cited_title":"Athey, G","cited_arxiv_id":null,"evidence_quote":"Shows recursive partitioning for heterogeneous treatment effects needs balanced treatment and control counts, the constraint FairUDT avoids by using probability differences."}],"review_version":1}