{"id":"f6af9c9d-b8bc-4072-aac5-cb739bf9bf16","arxiv_id":"2412.09896","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On the Adult income dataset, fairness libraries Fairlearn, AIF360, and What-If Tool all reduce measured gender disparity in a credit-style classifier, but the paper's comparison across libraries is not controlled.","lead":"This paper applies three fairness toolkits from Microsoft, IBM, and Google to the standard Adult income dataset to see how well each detects and reduces gender bias in a classifier. It reports that some built-in mitigation algorithms cut measured bias sharply while barely lowering accuracy, but the comparison is clouded by inconsistent baselines and missing error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison claim is not supported because the three libraries are evaluated under non-commensurable protocols (different metrics, different baselines, different models), so the paper's central quantitative conclusions cannot be drawn from the presented tables.","rationale":"The reader's weakest assumption identifies the same core problem: the paper's central comparison claim depends on three library evaluations being commensurable, but they are not. Section 5 explicitly describes different metrics for each library: Fairlearn uses accuracy and demographic parity difference; AIF360 uses Average Odds Difference and Statistical Parity Difference; What-If Tool uses accuracy and demographic parity difference. The baselines also disagree: Fairlearn baseline accuracy is 0.85, What-If baseline accuracy is 0.33, and AIF360 baseline metrics are 0.09 and 0.13. No explanation is given for why the same dataset and model would produce these different baselines, and no shared protocol, error bars, or reproducible code is provided despite the claim that 'codes and details can be found in [33]'. In addition, the percentage-improvement calculations are internally inconsistent: a change from 0.30 to 0.05 demographic parity difference is reported as 83% improvement (which is arithmetically a relative reduction of 83.3%), but a change from 0.09 to 0.007 AOD is reported as 8.3% better even though the relative reduction is 92.2%. This suggests the reported percentages are not computed on a consistent basis, undermining confidence in the quantitative headline. Given that the paper's stated objective is to compare the effectiveness of the libraries and its strongest claim is a quantitative comparison, the load-bearing concern is about the validity of that comparison. I do not see an independent, machine-checked proof, parameter-free derivation, or falsifiable prediction that would rescue the comparison. The conclusion that each tool has strengths and limitations is too generic to justify the headline numbers. Thus the reader's REJECT verdict remains appropriate, and no adjustment is needed.","tokens_in":7262,"tokens_out":2073,"duration_ms":17136,"concrete_test":"Re-run the three library evaluations on the same Adult dataset with the same train/test split, same XGBoost model, same sensitive attribute (gender), and a single agreed-upon fairness metric (e.g., demographic parity difference) and a single agreed-upon performance metric (accuracy), applying each library's documented mitigation algorithms. If the Fairlearn, AIF360, and What-If Tool evaluations still produce the same order of effectiveness (e.g., exponential gradient best with DPD 0.05), then the qualitative comparison survives; if the numbers change materially or the protocols cannot be aligned, the paper's central comparison claim is unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that the three fairness libraries can be compared for effectiveness (Objective 2, Section 1), and the strongest claim in Section 5 is that the exponential gradient algorithm achieved 0.85 accuracy (1.10% decline) and reduced demographic parity difference from 0.30 to 0.05 (83% improvement). The load-bearing assumption is that all three library evaluations are commensurable, but the paper violates that assumption. First, Section 5 uses different metrics per library: Fairlearn uses Demographic Parity Difference, AIF360 uses Average Odds Difference for performance and Statistical Parity Difference for fairness, and What-If Tool uses Demographic Parity Difference. The headline baseline also differs: the Fairlearn baseline accuracy is 0.85 and demographic parity difference is 0.30, while the AIF360 baseline SPD is 0.13 and the What-If Tool baseline accuracy is 0.33. These are not different measurements of the same quantity; they are different quantities measured under different protocols. Second, the What-If baseline accuracy of 0.33 directly contradicts the Fairlearn baseline accuracy of 0.85 for what should be the same Adult dataset and same Gradient Boosting model. The paper never explains this discrepancy. Third, percentage improvements are computed inconsistently: Section 5 reports 'an 83% improvement' for a change from 0.30 to 0.05 demographic parity difference, which is a 0.25 absolute reduction or 83.3% relative reduction, but also reports AIF360 changes like AOD 0.09 to 0.007 as '8.3% better' when the relative reduction is 92.2% and AOD 0.09 to 0.0001 as '9% better' when it is 99.9%. These numbers appear to be mislabeled or computed against a different baseline, making the quantitative comparisons internally untrustworthy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies three fairness toolkits—Fairlearn, AIF360, and the What-If Tool—to an Adult Income classification model built with XGBoost, using gender as the sensitive attribute. It reports baseline and post-mitigation accuracy and fairness metrics for each library, claims that each library has unique strengths and limitations, and singles out Fairlearn's exponentiated gradient algorithm as the best (accuracy 0.85, demographic parity difference 0.05, a 1.10% accuracy decline and an 83% fairness improvement over baseline). The paper concludes that the three libraries can be compared for effectiveness and offers practical guidance for practitioners. The quantitative support is undermined by inconsistent baselines across libraries, category errors in metric selection, arithmetic mistakes, and missing table content.","tokens_in":7634,"tokens_out":6575,"duration_ms":62216,"significance":"If the headline result were valid, it would be a useful practical data point: on the Adult dataset, exponentiated gradient reduces demographic parity difference from 0.30 to 0.05 with only a 1.10% accuracy decline. The paper also provides a broad literature review and makes its code available through a GitHub link. However, the central comparative claim is not supported by the experiments as reported. The inconsistent baselines and arithmetic errors mean that the paper does not currently establish its stated contribution of comparing the effectiveness of the three fairness libraries.","major_comments":[{"comment":"The three library evaluations are not commensurable, so Objective 2's comparison claim does not follow. Fairlearn is evaluated with accuracy and Demographic Parity Difference; AIF360 uses Average Odds Difference as the 'performance' metric and Statistical Parity Difference as the fairness metric; and the What-If Tool uses accuracy and Demographic Parity Difference at thresholds 0.2 and 0.4. The baselines also differ (accuracy 0.85/DPD 0.30 for Fairlearn; AOD 0.09/SPD 0.13 for AIF360; accuracy 0.33/DPD 0.19 for What-If). Because the metrics, models, and protocols differ across libraries, the reported numbers cannot be ranked to identify the 'best' algorithm.","section":"§5, Tables 1–3"},{"comment":"The What-If Tool baseline accuracy of 0.33 contradicts the Fairlearn baseline accuracy of 0.85 for what is described as the same Adult dataset and Gradient Boosting model. The manuscript offers no explanation for this discrepancy. At least one of the baselines is incorrect or the underlying models/pipelines are different, and all percentage improvements computed from these baselines are therefore unreliable.","section":"§5, Table 1 and Table 3"},{"comment":"Several reported percentage improvements are arithmetically incorrect. For Reweighing, AOD goes from 0.09 to 0.007 (roughly a 92% reduction, not 8.3%) and SPD from 0.13 to 0.01 (roughly 92%, not 12%). For Equalized Odds, the reductions are roughly 99.9% and 100%, not 9% and 13%. In the What-If Tool paragraph, accuracy increasing from 0.33 to 0.62 is about an 88% relative improvement, not 29%, and DPD decreasing from 0.19 to 0.01 is about a 95% reduction, not 18%. These errors affect the claim about which algorithm performed best.","section":"§5, AIF360 and What-If Tool results"},{"comment":"Average Odds Difference is a fairness metric, not a model performance metric; describing it as the selected 'performance metric' conflates predictive performance with group fairness. In addition, Equalized Odds is assigned to the post-processing stage but described as imposing constraints during model training; the manuscript should clarify whether EqualizedOddsPostprocessing or an in-processing method was used.","section":"§5, AIF360 paragraph"},{"comment":"The manuscript references Table 1, Table 2, and Table 3 as the evidence for the headline results, but no tabular content is included in the submitted text. Without the actual tables, the numerical claims in Section 5 cannot be verified or reproduced from the manuscript alone.","section":"Tables 1–3"},{"comment":"All results appear to be point estimates from a single 80-20 split with no repeated runs, cross-validation, or confidence intervals. The reported differences, such as the 1.10% accuracy decline and the 83% demographic-parity improvement, are therefore not established beyond sampling noise.","section":"§3 and §5"}],"minor_comments":[{"comment":"The thresholds 0.2 and 0.4 appear to be selected after inspecting the results, with no principled threshold-selection procedure or separate validation; this should be stated explicitly.","section":"§5, What-If Tool"},{"comment":"The XGBoost hyperparameter grid is not specified, so the grid-search tuning cannot be reproduced.","section":"§3"},{"comment":"Reference [12] points to a Kaggle code page; the canonical UCI Adult dataset citation should be provided instead.","section":"References"},{"comment":"There are frequent spacing artifacts in the text (e.g., 'F airlearn' and 'T ool') and the captions for Tables 1–3 are present but the tables themselves are absent; a careful editorial pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is an application-oriented comparison report rather than a research contribution with validated findings. The numerical inconsistencies (baseline accuracy 0.33 vs 0.85, arithmetic errors in percentage improvements, missing tables) are not local typographical issues, and the central comparative claim would require a new, controlled experimental design with consistent metrics, repeated runs, and complete tables. If the authors resubmit a fully corrected and expanded version, I would be willing to review it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one so you don't have to. It is a straightforward application of three fairness libraries (Fairlearn, AIF360, What-If Tool) to the Adult Income dataset, with the stated aim of comparing their effectiveness. The one thing to know: the comparison claim doesn't hold up because the three evaluations are not commensurable, and the paper's own numbers contradict each other. The What-If baseline accuracy is 0.33 while the Fairlearn baseline for the same dataset and model type is 0.85; no explanation is offered. That alone breaks the cross-library comparison.\n\nWhat the paper does well: it is a readable walkthrough of how each library can be used, and the authors posted code (no commit hash, but the repo is referenced). For a practitioner who has never touched these tools and wants a first look at the APIs and typical metrics, sections 3 and 4 are serviceable. The qualitative observation that each toolkit has its own strengths and limitations is true and worth repeating, even if it is not new.\n\nThe soft spots are substantial. First, the design is not controlled: each library is evaluated with a different fairness metric (Demographic Parity Difference vs. Average Odds Difference/Statistical Parity Difference), different mitigation algorithms, and different baselines. You cannot conclude one library is more effective when the quantities measured are not the same. Second, the numerical reporting is careless. The paper says the exponentiated gradient algorithm kept accuracy at 0.85 with only a 1.10% decline — if it stayed at 0.85, the decline is 0%. It calls a change in AOD from 0.09 to 0.007 an '8.3% better' result, which is actually a 92% relative reduction (or 8.3 percentage points); similarly for SPD. These are not minor typos, since the quantitative claims are the main substance. Third, the What-If thresholds (0.2 and 0.4) appear selected post hoc to produce favorable numbers, and no error bars or repeated runs are provided, so all numbers are point estimates from one split.\n\nThe paper does not deliver a research contribution: no new method, metric, or insight beyond what the toolkits' documentation already demonstrates. Its value, if any, is as a tutorial. For a research venue I would desk reject; the central claim is unsupported by the evidence as reported. If the authors are willing to reframe it as a reproducibility/tutorial note, fix the percentages, explain or remove the contradictory baseline, and use identical metrics across libraries, a workshop might be a reasonable home. But as it stands, I would not send it to peer review.","headline":"Routine three-toolkit walkthrough on Adult whose central library-effectiveness comparison is undercut by non-commensurable protocols and internally inconsistent baselines.","tokens_in":8147,"tokens_out":3469,"would_cite":false,"duration_ms":34906,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bias mitigation on a standard income classifier cuts the measured gender parity gap by 83% while keeping accuracy essentially unchanged.","keywords":["fairness metrics","bias mitigation","demographic parity","classification","Adult Income dataset","Fairlearn","AIF360","What-If Tool"],"falsifier":"Re-run the exact pipeline with one fixed metric (e.g., demographic parity difference), the same 80/20 split, the same XGBoost model, and gender as the sensitive attribute for all three libraries; if exponentiated gradient does not bring demographic parity difference near 0.05 with accuracy near 0.85, or if the conflict between the 0.33 and 0.85 baselines is confirmed, the paper's central claim fails.","tokens_in":7002,"feed_emoji":"⚖️","tokens_out":6937,"duration_ms":66694,"temperature":0.7,"pith_summary":"On a widely used income-prediction dataset, the paper tries to show that off-the-shelf fairness libraries can detect and remove most of a classifier's measured gender bias without giving up accuracy. With gender as the sensitive attribute, Fairlearn's exponentiated gradient algorithm reduces demographic parity difference from 0.30 to 0.05—an 83% improvement—while accuracy stays at roughly 0.85, a reported 1.10% relative decline. The paper also reports that AIF360's Reweighing and Equalized Odds techniques cut average odds and statistical parity differences close to zero, and that threshold adjustment in the What-If Tool lifts accuracy from 0.33 to 0.62 while shrinking demographic parity difference from 0.19 to 0.01. The intended upshot is practical: these tools can be combined across preprocessing, in-processing, and postprocessing to make classification models fairer in real workflows.","feed_headline":"Fairness fix cuts demographic parity gap by 83%","feed_subtitle":"On a standard income classifier, one mitigation keeps accuracy near 0.85 while nearly closing the gender gap.","key_machinery":"The paper's central machinery is a fairness-mitigation pipeline: a baseline XGBoost classifier on the Adult dataset, a sensitive attribute (gender), and a set of fairness metrics and mitigation algorithms keyed to three pipeline stages. The headline metric is demographic parity difference—the gap in positive-prediction rates between male and female groups—complemented by average odds difference and statistical parity difference for the AIF360 experiments. The algorithms doing the work are exponentiated gradient (an in-processing method that trains under fairness constraints via multiplicative-weight updates), threshold optimizer (a postprocessing method that sets decision thresholds to satisfy a fairness criterion exactly), Reweighing (a preprocessing method that reweights training samples by group representation), and Equalized Odds (a constraint forcing equal false-positive and true-positive rates across groups). The What-If Tool contributes interactive threshold adjustment, and the paper's argument is carried by comparing baseline versus mitigated values of these metrics.","core_discovery":"The paper's central claim is that bias mitigation on a structured classification task is both measurable and achievable with standard libraries, and that the three libraries are complementary rather than interchangeable. On the Adult dataset, the best single result is Fairlearn's exponential gradient: demographic parity difference drops from 0.30 to 0.05 with only a 1.10% accuracy decline relative to the 0.85 baseline; combining it with threshold optimizer gives similar results. AIF360's Reweighing and Equalized Odds bring average odds difference down to 0.007 and 0.0001 respectively, with statistical parity difference at 0.01 and 0.0, and the What-If Tool's threshold tuning improves both accuracy and demographic parity. The authors conclude that stage-wise mitigation—preprocessing adjustments followed by postprocessing constraints—can satisfy accuracy and fairness objectives simultaneously, and they frame the result as practical guidance for fairness tooling.","pith_inferences":["Across datasets beyond Adult, the same library stack could be benchmarked on a common metric to test whether the 83% parity reduction is typical or specific to this dataset's gender/income skew.","The paper's three-library comparison is not a controlled experiment because different fairness metrics, baselines, and algorithms are used; a unified re-run with identical metrics would tell which library actually performs best.","Near-zero parity after mitigation does not by itself show the model is fair in outcome quality; comparing error rates and calibration for male and female groups would test whether the fairness gain is substantive."],"forward_implications":["If the Fairlearn result holds, practitioners can remove most measured demographic parity difference on tabular income prediction while keeping accuracy essentially flat, making fairness mitigation cheap enough to run by default.","Combining preprocessing and postprocessing mitigations can match or exceed single-stage results, supporting the paper's recommendation of integrated pipelines.","AIF360's near-zero average odds and statistical parity differences after Reweighing and Equalized Odds show that group-level parity constraints can be imposed without large accuracy losses in this setting.","Threshold-based tuning with the What-If Tool gives a code-free lever for trading accuracy against parity, and in the paper's experiment it lifted both from a weak baseline."],"supporting_citations":[{"why":"Supplies the Adult Income dataset that the entire study runs on.","marker":"[12]"},{"why":"Defines equal opportunity and equalized odds, the fairness constraint used by AIF360's postprocessing.","marker":"[16]"},{"why":"Provides Fairlearn's metrics and the exponentiated gradient and threshold optimizer mitigation algorithms.","marker":"[30]"},{"why":"Provides AIF360's Reweighing and Equalized Odds implementations and fairness metrics.","marker":"[31]"},{"why":"Provides the What-If Tool for threshold-based performance and fairness exploration.","marker":"[32]"},{"why":"The project repository containing the code and details needed to reproduce the experiments.","marker":"[33]"},{"why":"The equality-of-opportunity result behind the threshold optimizer's exact satisfaction of fairness criteria.","marker":"[34]"}],"fun_headline_variants":["83% less bias on income classifier, accuracy barely moves","Cut demographic parity gap by 83% with Fairlearn","Fairness tools trade 1.1% accuracy for 83% parity gain","Adult dataset: 83% bias drop, accuracy intact","Bias down 83% on income model, accuracy steady"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison only works if the three library evaluations measure the same thing on the same footing; the paper's baselines and metrics differ across libraries, so if those setups are not equivalent, the comparison does not follow.","fun_headline_variants_meta":{"raw":{"variants":["83% less bias on income classifier, accuracy barely moves","Cut demographic parity gap by 83% with Fairlearn","Fairness tools trade 1.1% accuracy for 83% parity gain","Adult dataset: 83% bias drop, accuracy intact","Bias down 83% on income model, accuracy steady"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3542,"prompt_tokens":942,"completion_tokens":2600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2513}},"tokens_in":558,"tokens_out":2600,"duration_ms":20338,"temperature":1.0,"reasoning_tokens":2513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:35:29.088577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact pipeline with one fixed metric (e.g., demographic parity difference), the same 80/20 split, the same XGBoost model, and gender as the sensitive attribute for all three libraries; if exponentiated gradient does not bring demographic parity difference near 0.05 with accuracy near 0.85, or if the conflict between the 0.33 and 0.85 baselines is confirmed, the paper's central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the What-If Tool for threshold-based performance and fairness exploration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The project repository containing the code and details needed to reproduce the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Adult Income dataset that the entire study runs on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines equal opportunity and equalized odds, the fairness constraint used by AIF360's postprocessing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Fairlearn's metrics and the exponentiated gradient and threshold optimizer mitigation algorithms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides AIF360's Reweighing and Equalized Odds implementations and fairness metrics."},{"cited_title":"Equality of opportunity in supervised learning","cited_arxiv_id":null,"evidence_quote":"The equality-of-opportunity result behind the threshold optimizer's exact satisfaction of fairness criteria."}],"review_version":1}