{"id":"43c5ab47-c568-41ce-ad59-531f2e6c5d4f","arxiv_id":"2412.01585","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FairML.jl is a Julia package combining resampling, constrained optimization, and cut-off selection to reduce disparate impact and disparate mistreatment in binary classification.","lead":"This paper presents FairML.jl, a Julia package that applies three stages (data resampling, constrained optimization, and cut-off tuning) to make binary classifiers fairer with respect to sensitive attributes. The package extends fair logistic regression and SVM to mixed-effects data, but its empirical support comes only from synthetic simulations and refers to the authors' own prior papers for the core derivations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The simulations' 1% training split undermines the empirical basis for the claim that FairML reduces DI and DM.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and the reader's rationale already notes the 1% training split as a serious methodological concern. However, I would not single out the mixed-effects constraint derivation as the weakest load-bearing premise. Adding the random effect g_i to the linear predictor inside the Zafar-style constraints is a natural extension of the covariance-based construction, and the paper's prior work is the stated source. The more decisive weakness is the experimental protocol: a 1% training split is so unusual and so small that the reported improvements in DI and DM may not generalize, regardless of whether the mixed-effects constraints are theoretically sound. A standard-split rerun would settle whether the package's claimed benefits are real, making this the single most useful check. I therefore partially disagree with the reader's choice of weakest assumption while agreeing with the conditional verdict.","tokens_in":15610,"tokens_out":13367,"duration_ms":125129,"concrete_test":"Re-run the Section 6 experiments with a conventional 70/30 train-test split, using the same seeds and parameter settings, and recompute the DI/DM box plots for each stage and for the combined stages. If the qualitative reductions in DI/DM vanish or become non-significant, the current experimental protocol cannot support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim, including the statement in Section 6 that 'utilizing both strategies in conjunction yields superior outcomes compared to employing either in-processing or post-processing alone,' rests entirely on the Section 6 simulations. Those simulations explicitly set 'Train-Test split: Approximately 1% of the dataset was used for the training set, and 99% for the test set.' With such a tiny training sample, regression coefficients and, in the mixed models, random effects are estimated from very few observations; fairness constraints fitted on a handful of points can be satisfied by chance, and fairness metrics computed on the 99% held-out set will be noisy. The paper does not report the dataset size, so the actual training n is unknown. If the 1% split is not a typo, the box plots in Figures 1-8 do not substantiate the headline claim that the package's stages reduce disparate impact and disparate mistreatment. This concern is more immediately load-bearing than the mixed-effects derivation issue: even if the mixed-effects constraints were valid, the experiments would still be too fragile to confirm that they work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents FairML.jl, a Julia package that unifies three stages of fair classification: a preprocessing resampling method to reduce disparate impact, in-processing constrained optimization for logistic regression and SVM (including mixed-effects extensions), and a post-processing cut-off selection method. The authors report simulation studies on synthetic data, with and without group structure, showing improvements in disparate impact (DI) and disparate mistreatment (DM), and argue that combining in-processing and post-processing yields superior results. The paper also describes the package interface and documents hyperparameters.","tokens_in":15767,"tokens_out":3732,"duration_ms":30178,"significance":"The package fills a gap in the Julia ecosystem, and the mixed-effects extension is a distinctive feature relative to existing Python and R fair-ML packages. The paper provides a clear statement of the package's interface and documents hyperparameters, which lowers the barrier for practitioners. However, the central claim of effective unfairness reduction rests on synthetic-only simulations with a highly unusual 1% training split; without stronger empirical validation and a derivation of the mixed-effects constraints, the significance is not yet established. If these are addressed, the package could be a useful contribution to the fair-ML toolset.","major_comments":[{"comment":"The empirical claim that FairML reduces DI and DM, including the statement in Section 6 that 'utilizing both strategies in conjunction yields superior outcomes compared to employing either in-processing or post-processing alone,' rests on simulations that use approximately 1% of the dataset for training and 99% for testing. With such a tiny training sample, the coefficients and random effects are estimated from very few observations, fairness constraints can be satisfied by chance, and the metrics on the 99% held-out set are noisy. The manuscript does not report the absolute dataset size, so the actual training n is unknown. This concern is load-bearing because all headline results are based on these experiments. The authors should rerun the experiments with standard splits (e.g., 70/30 or repeated cross-validation), report dataset sizes, and show that the fairness improvements are stable.","section":"Section 6, 'Train-Test split'"},{"comment":"The mixed-effects fairness constraints are introduced by replacing β⊤xℓ with β⊤xij + g_i in the Zafar et al. constraints (Equations (5)–(8)), with no derivation. The paper cites Burgard and Pamplona (2024b) for 'a similar construction,' but the population-level fairness properties of this substitution are not demonstrated. Since the mixed-effects models are the paper's main novelty relative to existing fair-ML packages, the authors should either provide a derivation showing that these constraints control DI and DM in the presence of random effects, or include a formal statement of the conditions under which they do so. As written, a reader cannot verify that the mixed-effects fair classifiers actually deliver fair classifications.","section":"Section 4, 'Mixed Model'"},{"comment":"All numerical validation is performed on synthetic data generated by the authors using the same model class (logistic/SVM with the same prediction functions and random effects). This creates a circularity risk: the experiments can confirm that the solver works on data that exactly matches the model assumptions, but they do not demonstrate performance on real-world data with unknown structure. The authors should add at least one or two real benchmark datasets (e.g., UCI Adult, COMPAS, or similar) to validate the package's practical utility, or clearly state this as a limitation and temper the corresponding claims.","section":"Section 6, 'Numerical Results'"}],"minor_comments":[{"comment":"The phrase 'the β's = [−2.0; 0.4; 0.8; 0.5; 2.0]' mixes notation; use a consistent vector notation and clarify whether β4 corresponds to the sensitive feature in both regular and mixed settings.","section":"Section 6.1"},{"comment":"The phrase 'The in-processing phase consist of' should be 'consists of.'","section":"Abstract and Section 1"},{"comment":"The selection rule B = argmax_v(AC_v − f m_v) is not accompanied by an explanation of why subtracting the fairness metric (where lower is better) is the correct criterion; a brief justification would help.","section":"Section 5"},{"comment":"The figures are described in the text but not visible in the manuscript; ensure the final version includes readable box plots with labeled axes and legends, and report the exact quantiles plotted.","section":"Section 6"},{"comment":"The sentence 'we proposeFairML.jl a Julia package' is missing a comma or definite article.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's mixed-effects novelty is drawn from the authors' own prior works (Burgard and Pamplona 2024a,b); the current manuscript does not add an independent derivation. The 1% train-test split is unusual enough that the authors may have intended a different percentage; the editor should request a clarification. The manuscript also exhibits a pattern of self-citation for the core technical content, which is acceptable if the claims are genuinely from those works, but the present paper should make the dependency explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FairML.jl is a concrete, usable package that does something genuinely missing in the fair-ML ecosystem: it brings Zafar-style fairness constraints into mixed-effects logistic regression and SVM. That alone makes it worth a look for anyone doing fair classification on grouped or survey data. The three-stage pipeline (resampling, constrained in-processing, cutoff post-processing) is coherent, and the resampling method, while simple, is a reasonable undersampling variant. The post-processing cutoff rule is a standard heuristic. Credit where due: the paper is clearly written, the package documentation is promised, and the simulations are honest in the sense that they show the intended directional effects.\n\nThe soft spots are serious, though. The 1% training split is the biggest one. Training on 1% of the data and testing on 99% means the fitted coefficients and random effects come from a handful of points; fairness constraints fitted on a tiny sample can be satisfied by chance, and the fairness metrics on the 99% holdout are noisy. If that split is not a typo, the boxplots in Figures 1-8 simply do not substantiate the claim that the stages reduce DI and DM. The simulations are also entirely synthetic, generated from the same model class the package fits, and there is no comparison against Fairness.jl, Aequitas, or fairml in R. That matters for a software paper.\n\nThe mixed-effects extension, which is the main novelty, is not derived here. The paper adapts the Zafar constraints by adding g_i to the linear predictor and then cites the authors' own prior papers for the construction. That may be fine, but it means the key fairness claim is unsupported in this manuscript. The paper also says 'only the most relevant results are shown' and defers the rest to GitHub, without giving a repository URL in the text.\n\nNone of these flaws are fatal. They are all addressable: use a conventional split, add benchmark datasets, include the derivation or a clear pointer, show more results. As a software contribution, the package deserves a serious referee. I would send it to review, but with a clear message that the empirical section needs major rework.","headline":"FairML.jl is a useful Julia package with a real mixed-effects gap to fill, but the 1% training split in the simulations guts the empirical claims.","tokens_in":16333,"tokens_out":2328,"would_cite":false,"duration_ms":19924,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FairML.jl packages fair classification as three stages — data resampling, constrained fitting, and cut-off choice — and its simulations show each stage, and their combinations, reduce disparate impact and disparate mistreatment.","keywords":["fair classification","Julia package","disparate impact","disparate mistreatment","mixed-effects models","constrained optimization","resampling","cut-off selection"],"falsifier":"Fit the mixed-effects fair logistic regression or support-vector machine on synthetic grouped data with about 100 groups, random-effect variance comparable to the fixed effects, and sensitive-attribute prevalence varying across groups; then evaluate disparate impact and disparate mistreatment on a large held-out test set. If the attained metrics are no better than those of an unconstrained mixed model across most simulation runs, then the mixed-effects constraints do not enforce fairness in the population.","tokens_in":15364,"feed_emoji":"⚖️","tokens_out":16816,"duration_ms":123594,"temperature":0.7,"pith_summary":"FairML.jl is a Julia package that delivers fair classification as a three-stage pipeline: a resampling step that balances training data across sensitive groups and labels, an in-processing step that embeds fairness constraints directly into logistic-regression and support-vector-machine optimization, and a post-processing step that chooses a classification cut-off favoring the user's fairness metric. The paper claims each stage reduces the unfairness it targets — disparate impact (unequal rates of positive prediction across sensitive groups) or disparate mistreatment (unequal misclassification rates across sensitive groups) — and that combining the in-processing and post-processing stages beats either alone. Its distinctive contribution is an extension of the fairness constraints to mixed-effects models, so that classifiers for grouped data, as from surveys or clinical studies, can also be made fair. The claims rest on extensive simulations with synthetic data, for both ordinary and grouped-data settings.","feed_headline":"FairML.jl cuts classifier bias in three stages","feed_subtitle":"Resampling, fairness-constrained fitting, and cut-off tuning each help; stacking them helps most.","key_machinery":"The load-bearing object is the family of fairness constraints of Zafar et al. (2017), adapted to the package's objectives. For disparate impact, the constraint bounds the empirical covariance $n^{-1}\\sum_{\\ell}(s_\\ell-\\bar{s})\\beta^{\\top}x_\\ell$ between the sensitive feature and the decision score; for disparate mistreatment, it balances false-negative and false-positive rates across sensitive groups using sums of $\\min(0,\\beta^{\\top}x_\\ell)$ over the true-positive and true-negative subsets, each bounded by a tolerance $c$. The paper's mixed-effects extension replaces the score $\\beta^{\\top}x_{ij}$ with $\\beta^{\\top}x_{ij}+g_i$ inside every constraint. Around this core, the preprocessing stage resamples each sensitive-by-label cell down to the size of the smallest cell, with replacement and repeated $R$ times, keeping the run with the best fairness on the full training set, and the post-processing stage searches cut-offs from 0.01 to 0.99, picking the value that maximizes accuracy minus the fairness metric among those costing no more than 5% accuracy relative to the 0.5 cut-off.","core_discovery":"The central claim is that fairness in classification can be engineered at three independent points and that the three stages reinforce one another. The in-processing stage is the heart of the package: logistic regression and support-vector machines are reformulated as constrained optimization problems whose constraints enforce disparate-impact balance, through a bound on the covariance between the sensitive attribute and the decision score, or disparate-mistreatment balance, through equalized false-positive and false-negative rates built from $\\min(0,\\cdot)$ terms that isolate misclassified points, with a user-set tolerance $c$ mediating the accuracy–fairness trade-off. The paper's new step is to insert a group-level random effect $g_i$ into the decision score inside every constraint, yielding eight mixed-effects fair classification problems. Synthetic-data simulations show each phase improves the metric it targets, and the combined in-processing-plus-post-processing pipeline outperforms each stage on its own.","pith_inferences":["The mixed-effects constraints are asserted by substitution, putting $g_i$ into the linear predictor of the Zafar constraints, rather than derived, so whether the fairness guarantee survives at the population level is not established; a natural check is to use these constraints where group sizes and random-effect variance are extreme and the sensitive attribute is unevenly distributed across groups","Preprocessing is disabled for mixed models precisely because down-sampling can empty small groups; a weighting-based or synthetic-resampling preprocessing would let clustered data benefit from a preprocessing stage as well.","The post-processing rule of maximizing $\\mathrm{AC}_v - fm_v$ subject to a five-percent accuracy cap is a heuristic; the same machinery could be recast as an explicit accuracy–fairness Pareto frontier."],"forward_implications":["A user can target a specific fairness complaint — disparate impact, false-positive-rate disparity, false-negative-rate disparity, or disparate mistreatment — by choosing the corresponding in-processing constraints, in both regular and mixed-effects models.","Because the preprocessing resampling also lowers fairness metrics beyond the disparate impact it was designed for, it can serve as a generic fairness booster ahead of any classifier the package accepts, including user-supplied ones.","For grouped data, the mixed-effects fair logistic regression and support-vector machine are the package's main new contribution, since existing fair-classification packages do not handle random effects.","Combining the in-processing and post-processing stages yields superior fairness outcomes in the simulations, which is why the paper recommends running both together.","The post-processing cut-off search works with any fairness metric in the package and costs at most five percent accuracy by construction, so it is a low-risk add-on to any fitted classifier."],"supporting_citations":[{"why":"Supplies the core fairness constraints — the covariance bound for disparate impact and the misclassification-rate balance for disparate mistreatment — that the in-processing phase solves.","marker":"Zafar et al. (2017)"},{"why":"Supplies the $\\min(0,\\cdot)$ device that isolates false negatives and false positives, from which the FNR and FPR constraints are built.","marker":"Zafar et al. (2016)"},{"why":"Provides the mixed-effects support-vector machine and the 'similar construction' the paper cites as the basis for its mixed-effects fairness constraints.","marker":"Burgard and Pamplona (2024b)"},{"why":"Provides the mixed-effects logistic regression formulation that the package's grouped-data classifiers extend.","marker":"Burgard and Pamplona (2024a)"},{"why":"Defines equalized odds, the notion the paper operationalizes as disparate mistreatment and enforces through its DM constraints.","marker":"Hardt et al. (2016)"},{"why":"Defines disparate impact, the fairness notion targeted by the preprocessing stage and the DI constraints.","marker":"Barocas and Selbst (2016)"},{"why":"Supplies the direction-free disparate-impact metric $1-\\min(di,di^{-1})$ that the paper adopts for evaluating fairness.","marker":"Radovanović et al. (2020)"}],"fun_headline_variants":["Three-step FairML.jl beats single-stage fair classifiers","Mixed-effects fairness constraints make FairML.jl powerful","FairML.jl: resample, constrain, tune for fairness","Classify fairly: resampling, constrained fitting, cut-off tuning","Stacking fairness fixes works best in FairML.jl"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The package's headline novelty — fairness constraints for mixed-effects models — rests on the assumption that inserting a group-level random effect into the fairness constraints preserves the fairness guarantees, a step the paper asserts by analogy to an earlier paper rather than proves.","fun_headline_variants_meta":{"raw":{"variants":["Three-step FairML.jl beats single-stage fair classifiers","Mixed-effects fairness constraints make FairML.jl powerful","FairML.jl: resample, constrain, tune for fairness","Classify fairly: resampling, constrained fitting, cut-off tuning","Stacking fairness fixes works best in FairML.jl"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1506,"prompt_tokens":851,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":467,"tokens_out":655,"duration_ms":5586,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:29.249182+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the mixed-effects fair logistic regression or support-vector machine on synthetic grouped data with about 100 groups, random-effect variance comparable to the fixed effects, and sensitive-attribute prevalence varying across groups; then evaluate disparate impact and disparate mistreatment on a large held-out test set. If the attained metrics are no better than those of an unconstrained mixed model across most simulation runs, then the mixed-effects constraints do not enforce fairness in the population.","supporting_citations":[],"review_version":1}