{"id":"7a18ed24-c6aa-4b56-a271-0ff3c24f4116","arxiv_id":"2504.12587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GroupCART trains decision trees with weighted fairness and accuracy split criteria, selects the Pareto-best models on validation, and combines their votes to reduce bias in predictions.","lead":"This paper introduces GroupCART, a way to build decision-tree ensembles that consider both accuracy and fairness when choosing splits, then votes among the best trade-off models. The authors report fairer predictions with little accuracy loss across seven standard datasets, with adjustable user weights between performance and fairness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The d2h rankings in Tables 4–6 treat disparate impact (DI) as smaller-is-better, but Table 2 defines DI as a ratio where 1.0 is fair; this inverts the DI comparison and likely reverses the headline 'optimal trade-off' claim.","rationale":"The paper's central claim is that GroupCART achieves an optimal performance-fairness trade-off. That claim rests on the d2h summary scores in Table 6, computed from Eq. 4 with the metric optima implied by Table 6. The treatment of DI is internally inconsistent: Table 2 defines DI as P(Y=1|PA=0)/P(Y=1|PA=1), the fair value of which is 1 (or 100%), yet Table 5 says 'smaller is better' for all fairness metrics and the d2h calculation uses 0 as DI's optimum. Reproducing the Adult:Sex d2h for CART requires the DI term (90-0)^2, not (90-100)^2, confirming the 0-optimum interpretation. Under the correct orientation, GroupCART's low DI values (47, 39, 12, etc.) are signs of worse discrimination, not better, and the reported d2h rankings reverse on at least Adult:Sex and Adult:Race. The reader's weakest-assumption (FR-based early stopping possibly using test data, §6.3) is a genuine procedural flaw and should be checked, but the DI direction error is more directly load-bearing for the central empirical claim because it corrupts the paper's own summary metric across all datasets. The concern is testable analytically from Table 5. Thus the paper should not be accepted in its current form; a re-run with corrected DI orientation and validation-based early stopping would be needed to assess whether any version of the claim survives.","tokens_in":18635,"tokens_out":14175,"duration_ms":132973,"concrete_test":"Recompute Table 6 d2h from the median values in Table 5 with the DI term replaced by |DI - 100| (distance from the fair ratio) and all other terms unchanged. If GroupCART is no longer ranked first in a majority of the 8 settings (e.g., Adult:Sex and Adult:Race already flip), the central 'optimal trade-off' claim collapses and the paper must be re-evaluated with a correctly oriented DI.","verdict_should_be":"REJECT","load_bearing_attack":"Table 2 defines Disparate Impact as DI = P(Y=1|PA=0)/P(Y=1|PA=1), a ratio for which the fair value is 1 (100%). Table 5 nevertheless states 'for all fairness metrics, smaller is better,' and the d2h calculation in Eq. 4/Table 6 uses an optimum of 0 for DI (the Adult:Sex CART row's d2h=124 only matches a DI contribution of (90-0)^2=8100, not (90-100)^2=100). This reverses the semantics of DI: under the paper's own definition, a lower DI means the unprivileged group receives a smaller fraction of favorable outcomes, i.e., more discrimination. The reported results are inverted on this axis: in Adult:Sex, GroupCART's median DI is 47 versus CART's 90, so GroupCART is actually less fair on DI, yet it is scored as better. Recomputing just the Adult:Sex and Adult:Race d2h values with DI distance |DI-100| flips both rankings (Adult:Sex: CART ~86 vs GroupCART ~89; Adult:Race: CART ~90 vs GroupCART ~93 instead of the reported 80 vs 86). Since Table 6's 'GroupCART ranked first in 6/8 cases' is the main evidence for the conclusion that GroupCART 'can provide an optimal performance-fairness trade-off,' this mis-specified objective is a load-bearing error, not a stylistic choice. The validation/test leakage of the FR early-stopping rule in §6.3 is a separate, also valid concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GroupCART, an ensemble of fairness-aware decision trees in which each tree is built with a different weight ratio between information gain on the protected attribute (IGS) and information gain on the class label (IGC). Candidate trees are filtered by Pareto-based non-dominated sorting on validation data, and the surviving models vote on test instances. The authors evaluate GroupCART on seven fairness datasets against CART, EGR, Fax-AI, MAAT, and FairMask, reporting accuracy, F1, group fairness metrics (AOD, EOD, SPD, DI), and an individual fairness flip rate (FR). They claim that GroupCART achieves on-par or better predictive performance while substantially improving fairness, and that the method can provide an 'optimal performance-fairness trade-off' based on distance-to-heaven (d2h) scores.","tokens_in":19035,"tokens_out":6005,"duration_ms":59356,"significance":"The central idea, treating fairness as a configurable property of tree-based learners and using multi-objective optimization plus ensemble voting to navigate the fairness-performance trade-off, is relevant to the empirical software engineering community. The paper has notable strengths: it uses seven datasets, repeats experiments 20 times, compares with four baselines, applies non-parametric significance and effect-size testing, and makes code and data publicly available. If the empirical claims are established after correcting the issues below, this would be a useful contribution to fairness-aware ML software. However, the headline evidence for 'optimal trade-off' currently rests on a d2h computation that reverses the semantics of disparate impact, and the individual-fairness result is weakened by an early-stopping rule that uses the reported outcome as its own stopping criterion. These are load-bearing problems in the current manuscript.","major_comments":[{"comment":"The d2h computation treats DI = 0 as the optimum, but Table 2 defines Disparate Impact as DI = P(Y=1|PA=0)/P(Y=1|PA=1), for which the fair value is 1.0 (100%). The reported values in Table 6 are only consistent with an optimum of 0 for DI: for example, the Adult:Sex CART row has d2h = 124, which requires a DI contribution of (90-0)^2 = 8100 rather than (90-100)^2 = 100. Under the paper's own definition, a lower DI means the unprivileged group receives a smaller fraction of favorable outcomes, i.e., more disparity, not less. Recomputing Adult:Sex and Adult:Race with |DI-100| reverses both GroupCART-versus-CART comparisons. Because Table 6's 'GroupCART ranked first in 6/8 cases' is the main support for the conclusion that GroupCART provides an optimal performance-fairness trade-off, this metric-direction error is load-bearing. Please correct the optimum vector for DI, re-run all d2h and ranking analyses, and revise the Table 5 caption so that it does not state 'for all fairness metrics, smaller is better.'","section":"§5.2, Eq. (4), Table 6"},{"comment":"The early-stopping rule described in §6.3 grows the ensemble until the flip rate FR reaches zero, and the paper then reports FR = 0 as a favorable outcome for GroupCART in Table 5. The manuscript never states whether the FR used for stopping is computed on the validation set or the test set. If the test set is used, the reported perfect individual fairness is a leakage artifact and the selected ensemble sizes (5, 10, 20) are tuned to the reported outcome. If the validation set is used, the test FR = 0 is still not independent evidence, because the ensemble size was selected to drive FR to zero. Please specify exactly which data split feeds the stopping rule, and, if necessary, re-evaluate with a fixed ensemble size chosen before observing test FR.","section":"§6.3 and Table 5"},{"comment":"The domination definitions in Eq. (1)-(3) state that lower objective values y are preferred, but the objectives listed in §5.2 include accuracy and F1, for which higher values are better. The paper does not state that these metrics are negated or converted to error rates before NDSorting is applied in Algorithm 2. If raw accuracy and F1 are passed to a lower-is-better domination rule, the Pareto filter could rank poor-performance models as dominant, which would undermine the selection of candidate trees. Please specify the exact objective transformation used in the implementation of NDSorting.","section":"§4.3 and §5.2"},{"comment":"The row 'Default: Sex' in Table 4 refers to a dataset that is not listed in Table 1 and is not described in §6.1. In addition, the d2h values in Table 4 (0.46-1.14) are on a completely different scale from those in Table 6 (46-124), and the manuscript does not explain whether the two tables use different objective vectors, different normalizations, or different subsets of metrics. This makes the RQ1 result ('up to 51% improvement from optimization') difficult to interpret and reproduce. Please document the datasets and the exact metric vectors used for each d2h calculation.","section":"Table 4 and §6.1"}],"minor_comments":[{"comment":"The DI formula has a typo: it reads 'P(Y=1|PA=0]' with a closing bracket instead of a division slash; it should be 'P(Y=1|PA=0)/P(Y=1|PA=1)'.","section":"Table 2"},{"comment":"The caption contains a grammatical error: 'Presented in the paper be Cruz et al.' should read 'Presented in the paper by Cruz et al.'.","section":"Figure 2 caption"},{"comment":"The text says Table 5 reports median values over 20 runs, while Table 6 is described as reporting mean d2h scores; the paper should state clearly which statistic is used in each table and why.","section":"Tables 5 and 6"},{"comment":"The term 'ensemble size N' is used ambiguously: it sometimes refers to the number of initialized candidate trees and sometimes to the size of the final voting ensemble after Pareto filtering. Please define both quantities explicitly.","section":"Algorithm 2 and §6.3"}],"recommendation":"major_revision","confidential_remarks":"The DI-direction error in the d2h computation is potentially embarrassing but fixable; the authors should be required to recompute all d2h and ranking results with the correct DI optimum and to re-run the Scott-Knott analysis if rankings change. The FR early-stopping issue also needs a clear statement of which data split controls stopping, and if test data were used, a re-analysis with a pre-registered ensemble size. I do not see evidence of fabrication, and the public code and data availability are positive signals; the main concern is that the current evidence does not yet support the 'optimal trade-off' headline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kewen and colleagues propose GroupCART, an ensemble of fairness-aware CARTs built by sweeping IGS/IGC weights, selecting a Pareto frontier via continuous domination, and majority-voting the survivors. Each component is prior art—Kamiran, MAAT, Cruz—but the combination is new and the paper is straightforwardly described. Credit where due: the benchmark is broad (seven datasets, five fairness metrics, twenty runs), the raw group-fairness numbers in Table 5 often show real gains on AOD/SPD with modest accuracy cost, and the link to code is there. That part is worth taking seriously.\n\nThe problem is that the headline comparison is scored with a metric direction that appears to be wrong. Table 2 defines disparate impact as a ratio where 1.0 is fair, yet Table 5 says smaller is better and Eq. 4/d2h treat 0 as the optimum. I checked the arithmetic: the Adult:Sex CART d2h of 124 only comes out if DI=90 contributes (90-0)^2, not (90-100)^2. Under the paper's own definition, GroupCART's DI of 47 is less fair than CART's 90 on Adult:Sex, and redoing the d2h for Adult:Sex and Adult:Race flips the two rankings. Since 'GroupCART ranked first in six of eight cases' is the main evidence for the optimal-trade-off claim, this is a load-bearing error, not a presentational quibble.\n\nThe second soft spot is the FR early-stopping rule in §6.3. The paper says it doubles ensemble size until flip rate reaches zero, then reports FR=0 as a favorable outcome. It never states whether the stopping decision uses validation or test data. If test, the perfect individual-fairness results are leakage. The group-fairness results do not depend on this rule, so I do not think the whole paper is circular—but the individual-fairness claim is currently unverifiable.\n\nMinor: RQ1 uses d2h to both select and evaluate, and Table 5 shows medians without readable error bars or significance ranks. These are fixable. The anonymous GitHub link is unverified, so reproducibility is promised, not shown.\n\nBottom line: the method deserves a serious referee, not a desk reject. A revision that fixes the DI orientation, redoes the d2h tables, and specifies or removes the FR-stopping ambiguity could make this a solid empirical contribution. As it stands, the central claim should be treated as unproven, and I would not rely on the numerical rankings.","headline":"Plausible incremental ensemble method, but the headline 'optimal trade-off' leans on an inverted disparate-impact direction and an under-specified early-stopping rule.","tokens_in":19527,"tokens_out":4324,"would_cite":false,"duration_ms":44874,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GroupCART builds fairness into decision trees and keeps accuracy nearly intact.","keywords":["GroupCART","algorithmic bias","fairness-aware decision trees","ensemble learning","multi-objective optimization","Pareto frontier","group fairness","individual fairness"],"falsifier":"Re-run GroupCART on the Adult dataset with protected attribute 'sex', computing the flip rate used for early stopping only on the validation split, and check whether the ensemble still stops at 20 members and whether the held-out test flip rate remains zero. If the test FR is nonzero or the ensemble sizes change, the reported perfect individual-fairness results are leakage artifacts.","tokens_in":18411,"feed_emoji":"⚖️","tokens_out":8443,"duration_ms":81511,"temperature":0.7,"pith_summary":"This paper sets out to show that algorithmic bias in decision-tree models can be reduced during model construction rather than fixed afterward. It introduces GroupCART, an ensemble of fairness-aware decision trees in which each tree is built with a different weight on information gain from protected attributes. The candidate trees are filtered to the Pareto frontier of the performance-fairness trade-off and combined by majority voting. The paper reports that on seven fairness datasets this yields group-fairness metrics comparable to or better than four existing mitigation methods, with flip rates usually zero and only a small loss in accuracy or F1.","feed_headline":"GroupCART offsets decision-tree bias at tiny accuracy cost","feed_subtitle":"Fairness-aware trees, Pareto filtering, and majority voting match or beat five baselines on seven datasets","key_machinery":"The central mechanism is the fairness-aware decision tree (FDT), which replaces the usual information-gain split criterion with a weighted sum of $IG_C$, the information gain on the class attribute, and $IG_S$, the information gain on the protected attribute. GroupCART instantiates $N$ such trees with weight ratios $i/N$ versus $1-i/N$, evaluates all candidates on validation data, and applies non-dominated sorting under continuous domination, a softened dominance rule that gives each candidate a scalar loss, to retain the Pareto frontier. Majority voting over those frontier models produces the final prediction. The weight ratio is the hyperparameter that moves the ensemble along the fairness-performance spectrum, and the voting step is what turns several trade-off-optimal configurations into one model.","core_discovery":"The paper's central claim is that optimizing tree splits for two objectives at once--decreasing entropy in the target label and increasing entropy in protected attributes--and then ensembling the resulting Pareto-optimal configurations produces models that sit on a better performance-fairness frontier than plain CART and than prior mitigation methods. GroupCART builds $N$ candidate trees with distinct $IG_C / IG_S$ weight ratios, keeps the non-dominated ones under continuous domination, and averages their predictions. In the reported experiments it matches or beats EGR, Fax-AI, MAAT, and FairMask on most datasets, reduces flip rate to zero in nearly all settings, and gives best distance-to-heaven scores in several cases while keeping performance close to the unconstrained baseline. The authors conclude from this that algorithmic bias within decision tree models can be mitigated through multi-task fairness-aware learning.","pith_inferences":["If the early-stopping rule is validated as leakage-free, the reported flip rate of zero for nearly all settings would indicate that ensemble voting alone can enforce individual fairness without per-instance constraints; this could be tested by recording FR as a function of ensemble size on held-out data.","The same weight-ratio construction should transfer to random forests and AdaBoost-style ensembles; the paper suggests it but does not evaluate it, so a direct experiment would show whether the Pareto-frontier mechanism, not the specific tree learner, is what delivers the trade-off.","A natural stress test is to apply GroupCART to datasets with more than two protected-attribute values or to non-binary protected attributes, since the current experiments only cover binary protected attributes."],"forward_implications":["Fairness can be treated as a configurable property of tree-based models, so bias mitigation becomes a search over weight ratios rather than a separate repair step.","Ensembles with more than two members can improve the fairness-performance trade-off, suggesting that two-model ensembles leave room for improvement.","Users can dial a continuous weight between performance and fairness, producing a family of deployable models rather than a single compromise.","Multiple protected attributes can be handled at once, so a model can be tuned for fairness on sex and race simultaneously."],"supporting_citations":[{"why":"Introduces the fairness-aware split criterion that combines information gain on the class and protected attributes, which GroupCART parameterizes.","marker":"[39]"},{"why":"Supplies the exponentiated-gradient reduction baseline that GroupCART must match on fairness-constrained classification.","marker":"[11]"},{"why":"MAAT is the two-member ensemble baseline whose size GroupCART extends and whose performance-fairness results anchor the comparison.","marker":"[22]"},{"why":"FairMask is the baseline targeting individual fairness; its flip-rate results are the direct point of comparison for GroupCART's FR scores.","marker":"[50]"},{"why":"Provides the observation that fairness-performance Pareto frontiers emerge from hyperparameter configurations, motivating GroupCART's configuration view.","marker":"[26]"},{"why":"Defines the flip-rate metric used both for evaluation and for the early-stopping rule that sets ensemble sizes.","marker":"[19]"},{"why":"Sets the standardized preprocessing steps applied to the seven datasets before training.","marker":"[14]"}],"fun_headline_variants":["Dual-entropy trees slash bias with little accuracy loss","GroupCART: fairness-aware splits, near-zero cost","Two-objective tree ensembling beats bias baselines","GroupCART finds fairer models without data fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the early-stopping rule that fixes ensemble size at 5, 10, and 20--by stopping when flip rate reaches zero--uses only validation data; the paper never states which data split drives that rule, so the reported test flip rates may not be independent if the test set was involved.","fun_headline_variants_meta":{"raw":{"variants":["Dual-entropy trees slash bias with little accuracy loss","GroupCART: fairness-aware splits, near-zero cost","Two-objective tree ensembling beats bias baselines","GroupCART finds fairer models without data fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2701,"prompt_tokens":868,"completion_tokens":1833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1767}},"tokens_in":484,"tokens_out":1833,"duration_ms":13650,"temperature":1.0,"reasoning_tokens":1767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:28:25.481139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run GroupCART on the Adult dataset with protected attribute 'sex', computing the flip rate used for early stopping only on the validation split, and check whether the ensemble still stops at 20 members and whether the held-out test flip rate remains zero. If the test FR is nonzero or the ensemble sizes change, the reported perfect individual-fairness results are leakage artifacts.","supporting_citations":[{"cited_title":"In: 2010 IEEE international conference on data mining, IEEE, pp 869– 874","cited_arxiv_id":null,"evidence_quote":"Introduces the fairness-aware split criterion that combines information gain on the class and protected attributes, which GroupCART parameterizes."},{"cited_title":"In: International conference on machine learning, PMLR, pp 60–69","cited_arxiv_id":null,"evidence_quote":"Supplies the exponentiated-gradient reduction baseline that GroupCART must match on fairness-constrained classification."},{"cited_title":"In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 1122–1134","cited_arxiv_id":null,"evidence_quote":"MAAT is the two-member ensemble baseline whose size GroupCART extends and whose performance-fairness results anchor the comparison."},{"cited_title":"IEEE Transactions on Software Engi- neering","cited_arxiv_id":null,"evidence_quote":"FairMask is the baseline targeting individual fairness; its flip-rate results are the direct point of comparison for GroupCART's FR scores."},{"cited_title":"In: 2021 IEEE International Conference on Data Mining (ICDM), IEEE, pp 1036–1041","cited_arxiv_id":null,"evidence_quote":"Provides the observation that fairness-performance Pareto frontiers emerge from hyperparameter configurations, motivating GroupCART's configuration view."},{"cited_title":"(2018) Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias","cited_arxiv_id":null,"evidence_quote":"Sets the standardized preprocessing steps applied to the seven datasets before training."}],"review_version":1}