REVIEW 3 major objections 5 minor 46 references
Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Omitting protected attributes can substantially reduce disparate impact without a meaningful loss in accuracy.
desk verdict The empirical finding is real and worth engaging; the supporting theory needs repair before the advertised claims can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the $ε$-Rashomon set $R_ε(F)$: all classifiers whose accuracy is no less than a reference classifier $F$ minus $ε$. The paper's key quantitative mechanism is the distribution of estimated probabilities $p(x)$ near the $0.5$ decision threshold, encoded by the function $e(λ)$, the extra error incurred by flipping all predictions whose label probability is within $λ$ of $0.5$. This yields a tight upper bound on how much two near-optimal classifiers can disagree, and an upper bound on how much their disparate impact can differ that is inversely proportional to the smaller group's size. For logistic regression, the load-bearing mechanism is the sigmoid's steepness at the threshold: a protected-attribute coefficient $c_G < 0$ shifts all disadvantaged logits, and setting that coefficient to zero changes predictions only on the set $Q(c_G)$ of disadvantaged points with predictions in $[σ(c_G), 0.5]$, giving the accuracy and disparate-impact bounds of Proposition 6.
What would settle it
On a fixed logistic-regression task with a binary protected attribute, compare three models on the same data: the aware model, the unaware model retrained without the protected attribute, and the aware model with the protected coefficient set to zero. If the retrained unaware model's disparate-impact reduction is consistently outside the interval predicted by Proposition 6 from the aware model's $Q(c_G)$, or if its accuracy change exceeds $2σ(−c_G)P(X ∈ Q(c_G))$, then the theoretical mechanism does not explain the empirical result and the central claim would need a different argument.
Extended reading notes
Core claim
The paper establishes that an aware logistic-regression classifier can produce a larger disparate impact than the base-rate difference in the data, and that the corresponding unaware classifier—one that does not use the protected attribute—can stay within a small accuracy loss while removing a fixed fraction of that impact. The theoretical engine is the shape of the sigmoid: a fixed group coefficient $c_G < 0$ shifts every disadvantaged member's logit by a constant, and near the 0.5 decision threshold that constant can push many disadvantaged individuals below the threshold even though it has small effect on average probabilities. On real data the paper observes accuracy reductions under 1% with disparate-impact reductions between roughly 15% and 80% across datasets, model classes, and protected attributes. In a real-world use case modeled on an Austrian job-training allocation algorithm, dropping gender left AUROC essentially unchanged while narrowing the gender gap in high-risk assignments.
Load-bearing premise
The load-bearing premise is that the unaware model studied in the theory—the aware model with the protected group coefficient removed—matches the unaware model built by retraining without that attribute; the paper gives no proof that these two constructions coincide.
Editorial extensions
If this is right
- In tabular logistic-regression settings, deploying the unaware counterpart of an aware model can serve as a less discriminatory alternative, because accuracy stays within about one percent while disparate impact drops by 15% to 80% in the tested settings.
- The multiplicity bounds imply that models with equal accuracy can disagree most on points whose predicted label probability is near 0.5, so near-threshold observations are where fairness differences between models concentrate.
- For high-stakes profiling, such as allocating job-training programs, omitting gender can leave the ranking quality essentially unchanged while reducing the gender gap in who is denied training.
- The paper's framing suggests that model selection over the Rashomon set, rather than accuracy optimization alone, can be the locus of fairness, and that protected attributes deserve explicit justification before inclusion.
Reading between the lines
- The paper leaves implicit that the near-threshold mechanism should extend to other smooth classifiers with sigmoid-type outputs, so the empirical pattern likely generalizes beyond the two model classes tested.
- A testable extension the authors do not run: measure whether the realized disparate-impact reduction from retrained unaware models scales with the aware model's group coefficient $c_G$ and the mass of $Q(c_G)$; if it does not, the lower-bound mechanism is not the whole story.
- A natural next comparison, not made here, is Fairness through Unawareness against explicit interventions such as threshold shifts; combining a small threshold change with unawareness might yield even larger disparate-impact reductions than either alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper revisits Fairness through Unawareness (FtU). It argues that omitting a protected attribute from a predictive model can reduce disparate impact with little or no loss in accuracy, and it connects this observation to the model-multiplicity literature. The authors provide theoretical results: general upper bounds on model multiplicity and on changes in disparate impact between classifiers with similar accuracy, and a lower bound for logistic regression showing that an 'unaware' model obtained by zeroing the protected coefficient can reduce DI within a bounded accuracy budget. They substantiate the claim empirically on ACS Income, ACS Employment, and Swiss ALMP data with logistic regression and gradient-boosted trees, and they illustrate the implications on a job-training allocation scenario inspired by the Austrian AMS system. The advertised central claim is that FtU can reduce algorithmic discrimination without necessarily reducing accuracy, both theoretically and empirically.
Significance. If the central claim holds, the paper is practically valuable: it challenges the default assumption that adding protected attributes improves accuracy, and it connects FtU to the less-discriminatory-alternative literature. The empirical part is credible and reproducible: code is provided, and the results show accuracy reductions below 1% with DI reductions between 15% and 80% across many settings. The case study on job-training allocation is relevant and clearly presented. The theoretical part, however, is currently not load-bearing because of sign errors and missing arguments in Propositions 5 and 6; these need repair before the advertised 'both theoretically and empirically' claim is supported. With those repairs, this would be a solid contribution to algorithmic fairness and model multiplicity.
major comments (3)
- [Section 4.2, Eq. (24)-(27) and Appendix B.2 (proof of Proposition 6)] The proposition and its proof are internally inconsistent for c_G < 0. Q(c_G) in Eq. (24) is {x in B : f(x) in [sigma(c_G), 0.5]}, which for c_G = -0.86 is the interval of predicted probabilities [0.297, 0.5]. The proof's flipped-set Eq. (95), however, is written as {x in B : -c_G <= sigma^{-1}(f(x)) < 0}, which is empty when c_G < 0; the correct condition is c_G <= sigma^{-1}(f(x)) < 0. The numerical example then identifies Q with women whose predictions lie in [0.5, 0.70) = [sigma(-c_G), 0.5), which is the post-zeroing prediction interval, not the pre-zeroing interval of Eq. (24); hence the claimed 14.5-point DI reduction is not an instance of Eq. (26). In addition, the proof's calibration bound is stated as sigma(-c_G) <= P(Y|X in Q) <= sigma(c_G), which is reversed for c_G < 0, and Eq. (25) with the stated numbers (sigma(-0.86) approx 0.70, P(Q) = 6.9%) gives about 9.7 percentage points, not the claimed 0.7 percentage points, for the accuracy change.
- [Section 4.1, Proposition 5] The statement uses delta in the right-hand side max_{H in R_{epsilon+delta}(B)} without defining delta or relating it to F or epsilon. The proof constructs G> and G< by modifying F on U = {x : H(x) != B(x)}, but it never shows that either G# lies in R_epsilon(F); no accuracy calculation for F is supplied. Consequently the existence of a model in the epsilon-Rashomon set of F with the claimed DI change is not established. The lower bound may be repairable, but as written it is not load-bearing.
- [Section 4.2 vs. Section 5] Proposition 6 concerns an unaware classifier F' obtained by setting the protected coefficient of the fitted aware model to zero, while the empirical unaware models in Section 5 are retrained from scratch without the protected attribute. The paper does not argue that these two procedures produce the same or similar classifiers, so the lower bound in Proposition 6 does not necessarily explain the empirical DI reductions plotted in Figure 3. This is a gap between theory and experiments; the empirical existence claim survives, but the claimed theoretical explanation for the logistic-regression results is not currently established.
minor comments (5)
- [Table 4] The DBR column is labeled 'Disparate Benefit Ratio' but the values (0.15, 0.06, etc.) are differences in base rates as defined in Eq. (3); the label and definition should be corrected.
- [Section 3.2] The sentence 'even a moderate difference of 0.05 closer to the margins corresponds to a large difference of 0.18 near the threshold' is unclear; the 0.05 is a probability difference at the margins and the 0.18 is a probability difference at the threshold induced by the same logit offset; please rephrase.
- [Section 4.1, Eq. (15)] lambda_epsilon is defined with arg max, but ties are not addressed; state a tie-breaking rule or define it as a supremum/infimum.
- [Appendix B.2, Eq. (95)] Besides the sign issue noted in the major comments, the proof would benefit from explicitly naming the pre-sigmoid variable z = sigma^{-1}(f(x)) to avoid confusion between f and f' intervals.
- [Section 1 and Related Work] The claim of providing 'the first tight bounds on model multiplicity' should be qualified relative to Black et al. [7], whose bound is recovered in Corollary 1; the novelty is tightness under only the cumulative distribution of p(x), and this should be stated more precisely.
Circularity Check
No significant circularity: the empirical comparisons are direct and the theoretical bounds are derived from stated assumptions; only minor non-load-bearing self-citations appear.
full rationale
The paper's main existence claim is supported by direct experiments on public datasets: unaware models are retrained from scratch and compared to aware models on measured accuracy and disparate impact, so the empirical reductions are not constructed from the theoretical bounds. Propositions 2-6 are mathematical derivations from explicitly stated assumptions (0-1 loss, calibration, logistic sigmoid), and none of them fits a parameter to the result it announces. Proposition 6 derives the DI change of the coefficient-zeroed model F' from the definition of F' and the flipped set; this is a theorem, not a fitted prediction, although its numerical example contains interval/sign inconsistencies and the bound is not shown to govern the retrained unaware models used in Section 5. Those are correctness/validity concerns, not circularity. The self-citations ([26] on questioning data-generating distributions, [19] on race categorization) are contextual background and do not carry the derivation. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled via citation, and no existing empirical pattern is merely renamed. Hence no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Existence of an empirical joint distribution P over (X,Y) with binary groups A and B partitioning X.
- domain assumption Calibration as defined in Eq (4) of Section 3.1.
- domain assumption The 0.5 classification threshold, with classifier F(x) = 1[f(x) >= 0.5].
- domain assumption For Proposition 6, the aware logistic regression model is calibrated on Q(c_G), or at least the label ratio lies in [sigma(c_G), sigma(-c_G)].
- standard math Bayes-optimal classifier and 0-1 loss define the Rashomon sets.
Cite this review
Pith. "Pith review of Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity." pith.science (2026). https://pith.science/paper/RZCZQOW3
@misc{pith2026250516638,
author = {Pith},
title = {Pith review of: Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity},
year = {2026},
howpublished = {\url{https://pith.science/paper/RZCZQOW3}},
note = {Machine review of arXiv:2505.16638}
}
read the original abstract
Fairness through Unawareness (FtU) describes the idea that discrimination against demographic groups can be avoided by not considering group membership in the decisions or predictions. This idea has long been criticized in the machine learning literature as not being sufficient to ensure fairness. In addition, the use of additional features is typically thought to increase the accuracy of the predictions for all groups, so that FtU is sometimes thought to be detrimental to all groups. In this paper, we show both theoretically and empirically that FtU can reduce algorithmic discrimination without necessarily reducing accuracy. We connect this insight with the literature on Model Multiplicity, to which we contribute with novel theoretical and empirical results. Furthermore, we illustrate how, in a real-life application, FtU can contribute to the deployment of more equitable policies without losing efficacy. Our findings suggest that FtU is worth considering in practical applications, particularly in high-risk scenarios, and that the use of protected attributes such as gender in predictive models should be accompanied by a clear and well-founded justification.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Doris Allhutter, Florian Cech, Fabian Fischer, Gabriel Grill, and Astrid Mager. 2020. Algorithmic profiling of job seekers in Austria: how austerity politics are made effective. Frontiers in Big Data (2020), 5
work page 2020
-
[2]
Ruben L Bach, Christoph Kern, Hannah Mautner, and Frauke Kreuter. 2023. The impact of modeling decisions in statistical profiling. Data & Policy 5 (2023), e32
work page 2023
-
[3]
Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities . MIT Press
2023
-
[4]
Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. California Law Review 104 (2016), 671
work page 2016
-
[5]
Emily Black, John Logan Koepke, Pauline Kim, Solon Barocas, and Mingwei Hsu. 2024. Less discriminatory algorithms. Georgetown Law Journal 113, 1 (2024)
work page 2024
-
[6]
Emily Black, Logan Koepke, Pauline Kim, Solon Barocas, and Mingwei Hsu. 2024. The legal duty to search for less discriminatory algorithms. arXiv preprint arXiv:2406.06817 (2024)
work page Pith review arXiv 2024
-
[7]
Emily Black, Manish Raghavan, and Solon Barocas. 2022. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 850–863
work page 2022
-
[8]
Danah Boyd and Kate Crawford. 2012. Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, Communication & Society 15, 5 (2012), 662–679
work page 2012
Show all 46 references
-
[9]
Leo Breiman. 2001. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science 16, 3 (2001), 199–231
2001
-
[10]
Toon Calders and Sicco Verwer. 2010. Three naive bayes approaches for discrimination-free classification. Data mining and knowledge discovery 21 (2010), 277–292
2010
-
[11]
Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163
2017
-
[12]
Madison Coots, Soroush Saghafian, David Kent, and Sharad Goel. 2023. Reevaluating the role of race and ethnicity in diabetes screening. arXiv preprint arXiv:2306.10220 (2023)
2023 arXiv
-
[13]
Sam Corbett-Davies, Johann D Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel. 2023. The measure and mismeasure of fairness. The Journal of Machine Learning Research 24, 1 (2023), 14730–14846
2023
-
[14]
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. InProceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining . 797–806
2017
-
[15]
Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021. Characterizing fairness over the set of good models under selective labels. In International Conference on Machine Learning . PMLR, 2144–2155
2021
-
[16]
André Cruz and Moritz Hardt. 2024. Unprocessing Seven Years of Algorithmic Fairness. In The Twelfth International Conference on Learning Representations
2024
-
[17]
Sam Desiere, Kristine Langenbucher, and Ludo Struyven. 2019. Statistical profiling in public employment services. An international comparison. OECD Social, Employment and Migration Working Papers 224 (2019)
2019
-
[18]
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. 2021. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems 34 (2021), 6478–6490. 16 Benedikt Höltgen and Nuria Oliver
2021
-
[19]
Miriam Doh, Benedikt Höltgen, Piera Riccio, and Nuria M Oliver. 2025. Position: The categorization of race in ml is a flawed premise. InForty-second International Conference on Machine Learning
2025
-
[20]
Sina Fazelpour and Zachary C Lipton. 2020. Algorithmic fairness from a non-ideal perspective. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 57–63
2020
-
[21]
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining . 259–268
2015
-
[22]
Jutta Gamper, Günter Kernbeiß, and Michael Wagner-Pinter. 2020. Das Assistenzsystem AMAS. Zweck, Grundlagen, Anwendung
2020
-
[23]
Talia B Gillis, Vitaly Meursault, and Berk Ustun. 2024. Operationalizing the Search for Less Discriminatory Alternatives in Fair Lending. In The 2024 ACM Conference on Fairness, Accountability, and Transparency . 377–387
2024
-
[24]
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016)
2016
-
[25]
Jürgen Holl, Günter Kernbeiß, and Michael Wagner-Pinter. 2018. Das AMS-Arbeitsmarktchancen-Modell. Arbeitsmarktservice Österreich, Wien (2018)
2018
-
[26]
Benedikt Höltgen and Robert C Williamson. 2025. We should avoid the assumption of data-generating probability distributions in social settings. arXiv preprint arXiv:2407.17395 (2025)
2025 arXiv
-
[27]
Christoph Kern, Ruben Bach, Hannah Mautner, and Frauke Kreuter. 2024. When small decisions have big impact: fairness implications of algorithmic profiling schemes. ACM Journal on Responsible Computing 1, 4 (2024), 1–30
2024
-
[28]
Matthias Kuppler, Christoph Kern, Ruben L Bach, and Frauke Kreuter. 2022. From fair predictions to just decisions? Conceptualizing algorithmic fairness and distributive justice in the context of data-driven decision-making. Frontiers in sociology 7 (2022), 883999
2022
-
[29]
Michael Lechner, Michael Knaus, Martin Huber, Markus Frölich, Stefanie Behncke, Giovanni Mellace, and Anthony Strittmatter. 2020. Swiss Active Labor Market Policy Evaluation [Dataset]. Distributed by FORS (2020). doi:10.23662/FORS-DS-1203-1
2020 doi
-
[30]
Robert Lipp. 2005. Job Seeker Profiling. The Australian Experience (2005)
2005
-
[31]
Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive multiplicity in classification. In International Conference on Machine Learning . PMLR, 6765–6774
2020
-
[32]
Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency. PMLR, 107–118
2018
-
[33]
Emanuel Moss. 2022. The objective function: Science and society in the age of machine intelligence. arXiv:2209.10418 (2022)
2022 arXiv
-
[34]
Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 560–568
2008
-
[35]
Juan Carlos Perdomo. 2024. The relative value of prediction in algorithmic decision making. In Proceedings of the 41st International Conference on Machine Learning. 40439–40460
2024
-
[36]
Juan C Perdomo, Tolani Britton, Moritz Hardt, and Rediet Abebe. 2023. Difficult Lessons on Social Prediction from Wisconsin Public Schools. arXiv:2304.06205 (2023)
2023 arXiv
-
[37]
Theodore M Porter. 1995. Trust in numbers. Princeton University Press
1995
-
[38]
Four-Fifths Rule
Manish Raghavan and Pauline T Kim. 2024. Limitations of the" Four-Fifths Rule" and Statistical Parity Tests for Measuring Fairness. Geo. L. Tech. Rev. 8 (2024), 93
2024
-
[39]
Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. 2024. Amazing things come from having many good models. arXiv preprint arXiv:2407.04846 (2024)
2024 arXiv
-
[40]
Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency . 59–68
2019
-
[41]
Lesia Semenova, Harry Chen, Ronald Parr, and Cynthia Rudin. 2023. A path to simpler models starts with noise. Advances in Neural Information Processing Systems 36 (2023)
2023
-
[42]
Amartya Sen. 2010. The idea of justice . Penguin books
2010
-
[43]
Vibhhu Sharma and Bryan Wilder. 2025. Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources. The Thirteenth International Conference on Learning Representations (2025)
2025
-
[44]
Ali Shirali, Rediet Abebe, and Moritz Hardt. 2024. Allocation Requires Prediction Only if Inequality Is Low. In Proceedings of the 41st International Conference on Machine Learning . 45114–45153
2024
-
[45]
Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. 2022. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. https://papers.ssrn.com/abstract=4238015
2022
-
[46]
Sebastian Zezulka and Konstantin Genin. 2024. From the Fair Distribution of Predictions to the Fair Distribution of Social Goods: Evaluating the Impact of Fair Machine Learning on Long-Term Unemployment. InThe 2024 ACM Conference on Fairness, Accountability, and Transparency ....
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.