Understanding Fairness-Accuracy Trade-offs in Machine Learning Models: Does Promoting Fairness Undermine Performance?

Junhua Liu; Kwan Hui Lim; Roy Ka-Wei Lee

REVIEW 5 major objections 5 minor 18 references

Understanding Fairness-Accuracy Trade-offs in Machine Learning Models: Does Promoting Fairness Undermine Performance?

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Machine learning models make more consistent admissions decisions than human experts.

desk verdict A plausibly written paper whose central ML-vs-human consistency comparison is likely a smoothing artifact, since the ML models are trained on the very human labels they are compared against. read the letter →

arxiv 2411.17374 v2 pith:LXJEZ7LF submitted 2024-11-26 cs.CL cs.AIcs.IR

Junhua Liu , Roy Ka-Wei Lee , Kwan Hui Lim This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords individualfairnessconsistencyscoreuniversityadmissionsBERTembeddingsmachinelearninghumandecision-makingfairness-accuracytrade-offBi-LSTM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether promoting fairness in machine-learning admission models must come at the cost of accuracy, and answers no: it measures individual fairness with a consistency score and finds that the XGB and Bi-LSTM models separate similar applicants more consistently than human evaluators do at three decision points. On a real dataset of 870 university applicant profiles, Bi-LSTM reaches 80.73% consistency on admission recommendations and 77.97% on offers, against human scores of 56.32% and 60.23%. The paper reports that ML consistency exceeds human consistency by margins of 14.08% to 18.79%, and argues that ML models can serve as decision-support tools to flag inconsistent human judgments while keeping accuracy at human-comparable levels.

What carries the argument

The consistency score, defined as $C = 1 - \frac{1}{N}\sum_{i=1}^N \left| \hat{y}_i - \frac{1}{k} \sum_{j \in \text{knn}(i)} \hat{y}_j \right|$, measures how similarly a classifier treats each applicant and its $k$ nearest neighbors, with $k=5$. The nearest neighbors are computed in a BERT embedding space: each applicant's four textual features are encoded with bert-base-uncased, the [CLS] token gives a 768-dimensional vector per feature, and concatenation yields a 3840-dimensional profile vector from which a feature-reranked similarity matrix is built. This similarity matrix is the machinery that makes the fairness comparison between humans and models possible, because it fixes a common notion of 'similar applicant' for both.

What would settle it

If a sample of human evaluators, shown pairs of applicants selected as k-nearest neighbors by the BERT embeddings, do not rate them as similarly qualified, then the consistency score ceases to be a measure of individual fairness as humans understand it, and the ML advantage could be an artifact of the embedding space.

Watch

Extended reading notes

Core claim

The central claim is that individual fairness, measured as decision consistency among similar applicants, is substantially higher for machine learning models than for human admission reviewers, and this fairness gain does not hurt accuracy. On the 870-profile admissions dataset, Bi-LSTM attains 80.73% consistency for admission recommendations and 77.97% for offers, compared with human scores of 56.32% and 60.23%, while its accuracy (82.76%) actually exceeds human accuracy at those stages (80.87% at AR). The paper interprets the human results as evidence of cognitive bias that grows in later decision stages, and proposes a hybrid pipeline where ML consistency checks prompt re-review of divergent cases.

Load-bearing premise

The consistency score assumes that similarity among applicants is accurately captured by distances in a BERT embedding space, so that the k-nearest neighbors in that space really are the 'similar individuals' whose treatment should match.

Editorial extensions

If this is right

If ML models are more consistent than humans at similar profiles, using them as an initial screening stage in admissions could reduce unwanted variability without sacrificing accuracy.
Disagreements between a model's decision and a human reviewer's decision can be flagged for additional review, targeting exactly the cases where human bias is most likely.
The consistency score gives admissions offices a quantifiable fairness metric beyond accuracy, letting them audit decision processes over time.
The finding that human consistency drops from shortlisting to later stages suggests biases enter when decision criteria become more complex, so interventions should focus there.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The general claim likely depends on the specific dataset and embedding; we would not extrapolate to other domains such as hiring or lending without similar experiments, since the BERT similarity space may not match each domain's fairness-relevant features.
The consistency metric as defined treats predictions as binary and averages over nearest neighbors; a finer-grained ordinal decision scale or a fairness definition based on ground-truth outcomes rather than predicted labels would likely change the absolute scores, though the relative human-model gap might persist.
A practical testable extension is to run the same consistency measurement on a dataset several times larger to see whether the ML advantage shrinks as human reviewers gain more experience with the applicant pool.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's headline result—that ML models are 14–19% more consistent than human evaluators—probably doesn't survive contact with its own methodology. The models are trained to predict the human decisions (Section 3.3), and their consistency scores are computed on those fitted predictions. Human consistency is computed on the raw labels. A regularized model fitted to noisy labels will always look smoother than the labels themselves, so the reported gap may just be the difference between a smoothed reconstruction and the original noise. The paper never tells us whether human consistency is computed on the same ten-percent test split used for the ML scores, which makes the comparison apples-to-oranges even in principle.

That said, the paper has real virtues. It applies a sensible, established consistency metric (Zemel et al. 2013) to a real admissions dataset, and it is clearly written. The idea of measuring individual fairness as local smoothness in a feature space is appropriate, and the accuracy numbers for the ML models are reported transparently. The empirical exercise is straightforward, and the authors are honest about the lack of code/data.

The soft spots are load-bearing, not minor. The lack of an independent human baseline is the central flaw; without repeated decisions by multiple raters, "human consistency" is just outcome noise. The title promises a fairness-accuracy trade-off analysis, but the paper merely lists both metrics; no trade-off is actually modeled or tested. Error bars and significance tests are missing throughout, so we have no sense of whether the 14–19% margin is even statistically distinguishable from zero. The KNN model's low accuracy (68.97%) muddies the comparison, and the embedding-based similarity space is not validated against any human notion of applicant similarity.

This is a workshop-level paper with a good question and a flawed answer. It could be useful as a cautionary example of how not to compare human and ML decision consistency, but it is not evidence that ML is fairer than humans. A serious referee would likely ask for major revisions or reject it outright, and I would not want to cite it as a factual result.

Recommendation: do not accept for peer review as is; reject, but encourage the authors to redo the analysis with multiple raters and a proper held-out evaluation.

Referee Report

5 major / 5 minor

Summary. The paper studies individual fairness in university admissions using 870 applicant profiles from an existing dataset. It introduces a consistency score (Eq. 1) based on k-nearest neighbors in BERT embedding space, applies it to human decisions at three stages (shortlisting, admission recommendation, offer) and to three ML models (KNN, XGB, Bi-LSTM), and reports that XGB and Bi-LSTM exceed human consistency by roughly 14 to 19 percentage points while maintaining comparable accuracy. The authors conclude that ML models could support a hybrid human-ML admissions pipeline by providing more consistent treatment of similar applicants.

Significance. If the headline result were valid, the paper would make a useful empirical contribution: it applies individual fairness to a realistic admissions task, uses actual human decision stages, and proposes a simple and transparent consistency metric in Eq. (1). The comparison of multiple model families and the use of contextual embeddings are also reasonable starting points. However, the central comparison is currently not an apples-to-apples measurement: human consistency is computed on raw human decision labels, while model consistency is computed on predictions of classifiers fitted to those same labels, so a large part of the reported gap can be explained by smoothing of label noise. The paper also lacks uncertainty quantification, omits KNN consistency scores, and does not validate the similarity space on which the fairness definition depends. These issues are load-bearing for the paper's main claim.

major comments (5)

[§3.3, §4.4, Eq. (1)] The central claim that ML models are more individually fair than humans is not established because the human and ML consistency scores measure different objects. Human C(AR)=0.5632 and C(OF)=0.6023 are computed directly on raw human decisions, whereas ML C(AR)=0.8073 and C(OF)=0.7797 are computed on predictions of classifiers trained to reproduce those same human labels. Any regularized classifier fitted to noisy labels smooths label noise, so its nearest-neighbor consistency in the embedding space will tend to exceed the raw labels' consistency even if the underlying decision process is identical. The paper never states that human consistency is computed on the same 10% test subset used for model evaluation, and no repeated decisions or multiple-rater data are described that could serve as an independent human consistency baseline. This is a load-bearing issue for RQ2 and for the abstract's headline margin.
[§4.3, Eq. (1)] The consistency score's validity depends entirely on the choice of similarity space and on k, since Eq. (1) uses k-nearest neighbors in the BERT embedding space to decide which applicants are "similar." The paper does not validate this embedding-based similarity against any admissions-relevant notion of similarity, and it reports no sensitivity analysis for k or for the embedding construction. Without such validation, the reported consistency scores are not necessarily measurements of individual fairness, because a different embedding or k could mechanically change every score in Table 1.
[Table 1, §4.4] Table 1 reports no consistency scores for the KNN model, yet Section 4.4 states that "both machine learning models demonstrate substantially higher consistency scores" after presenting three ML models. If KNN's consistency is unavailable, all claims about ML models must be restricted to XGB and Bi-LSTM; if it is available, it should be reported, since the retrieval-based KNN classifier is directly relevant to the k-nearest-neighbor definition in Eq. (1).
[§4.4, §3.1] All consistency and accuracy results are single point estimates from one 80/10/10 split of 870 profiles, with no error bars, bootstrap intervals, or significance tests. Because the raw classification metrics are close between humans and models (for example, Bi-LSTM accuracy 82.76% vs. human AR accuracy 80.87%), the reported 24.41 percentage-point consistency gap could be an artifact of the particular split, and the paper provides no evidence to rule this out.
[§4.4, RQ3] The accuracy comparison in RQ3 conflates two different targets. The ML models are trained to predict the human decision labels defined in Section 3.3, so their accuracy measures agreement with human decisions, not performance on the final admission outcome Type. The human decision rows in Table 1 are also evaluated against Type, so statements such as "Bi-LSTM ... even surpassing human decisions in accuracy" compare model accuracy on human labels with human accuracy on final offers, which are different quantities. This undermines the claim that ML models "outperform human in making admission decisions."

minor comments (5)

[§4.5] The text states there is "a gradual decrease in consistency from OF to AR stages, i.e. from 60.23% to 56.32%" and then says biases become more pronounced in later stages; since OF is later than AR and has higher consistency, this interpretation moves in the wrong direction.
[§4.5, Table 1] The sentence "early stages ... demonstrate strong classification performance i.e. over 8% in F1-Score and accuracy" should say "over 80%," since the reported values are 0.81 to 0.85.
[Abstract, §4.5] The margins 14.08% to 18.79% are absolute percentage-point differences, not relative improvements; for example, 80.73% versus 56.32% is a 43.3% relative increase. The paper should phrase these as percentage points or report relative changes.
[References] Reference 14 lists "Unknown, A." as the author; this placeholder must be completed before publication.
[§3.2] The described vector dimension needs clarification: four features with 768-dimensional [CLS] embeddings would concatenate to 3072 dimensions, while the paper reports 3840 dimensions, presumably because the concatenated Combined document is also included as a fifth feature; this should be stated explicitly.

Circularity Check

2 steps flagged · score 7.0 of 10

ML fairness advantage is a smoothing artifact: the models are trained on the same human decisions used as the human consistency baseline.

fitted input called prediction [Section 3.3 (Human Decisions)]
"We use three human decision points, such as Shortlisting (SL), Admission Recommendation (AR) and Offer (OF), as targets to assess prediction accuracy and consistency."

The human decisions are explicitly the training targets for the ML models. The same decisions are then used as the 'human' baseline in Table 1: the human consistency values (56.32% for AR, 60.23% for OF) are computed directly on these labels. Thus the ML consistency is measured on functions fitted to the very labels against which human consistency is measured, so any smoothing by the model is counted as a fairness gain. The paper also does not state that the human consistency scores are computed on the same 10% test split used for the ML models, so the comparison may even mix different subpopulations.
fitted input called prediction [Section 4.1 Eq. (1) and Section 4.4 (Fairness Consistency)]
"Formally, given a set of N applicants, we define the consistency score C of a classifier as: C = 1 − 1/N Σ_i |ŷ_i − 1/k Σ_{j∈knn(i)} ŷ_j|"

For the human rows of Table 1, ŷ is the observed human label; for the ML rows, ŷ is the model's prediction. Since each model is trained to reproduce those human labels, its ŷ values are smoothed reconstructions of the same data. Equation (1) therefore measures the local smoothness of the fitted model surface, not an independent property of human versus machine decision-making. The claimed 14.08–18.79% margin is the expected gap between a regularized function fitted to noisy labels and the noisy labels themselves, so the central fairness conclusion is forced by the training/evaluation construction rather than by an independent human-comparator measurement.

full rationale

The comparison is not formally circular in the narrow sense that Eq. (1) is misapplied to one side only: the formula is applied to both human labels and model predictions. However, the central claim that ML models are fairer than humans by 14.08–18.79% reduces to a fitted-input comparison. Section 3.3 declares the human decisions (SL/AR/OF) to be the targets for training and evaluation, and Section 4.1 defines consistency on a classifier's ŷ. For humans, that ŷ is the raw observed label; for ML, it is the output of a model fitted to those labels. Any regularized classifier fitted to noisy labels will tend to be more locally smooth than the labels themselves, because fitting averages over label noise. The paper provides no independent human baseline, such as repeated ratings by the same or multiple experts or held-out human decisions, against which the ML predictions could be compared. Consequently, the reported consistency gap largely quantifies the smoothing effect of fitting to the human labels, not a measured superiority of machines over humans in fairness. The dataset is introduced in a self-citation [8], but that is not the load-bearing circular step; the bias is in the human-vs-ML consistency comparison. The unvalidated BERT embedding space is a separate validity concern, not a circularity. Score 7 is appropriate because the qualitative direction of the result is statistically expected from the setup, even though the exact consistency values and accuracy numbers are empirical.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the consistency metric as a fairness measure, the meaningfulness of the BERT similarity space, the appropriateness of the training setup, and the reliability of the private dataset. The ML models appear to be trained on the human decisions they are compared against, so the consistency advantage may be an artifact of fitting a smooth function to noisy labels.

free parameters (3)

k in consistency metric = 5
Chosen by hand to balance local similarity and statistical stability; consistency scores change with k.
k in KNN classifier = 5
Selected empirically as the number of neighbors for the retrieval-based model.
XGB and Bi-LSTM hyperparameters = not reported
Tuned via randomized search on the validation set, but final values are not given in the paper.

assumptions (4)

domain assumption The consistency metric from Zemel et al. (2013) is a valid operationalization of individual fairness.
The paper imports this metric without justifying that consistency in predictions implies fairness.
domain assumption The BERT embedding space captures applicant similarity relevant to admissions.
Nearest neighbors are computed in this space for the consistency scores.
domain assumption The human decision labels are an appropriate target for training and comparing ML models.
If the ML models are trained on these labels, the comparison is partly circular; the paper does not specify the training target clearly.
domain assumption The private dataset from the authors' prior work is representative and the human annotations are reliable.
No details on rater demographics or annotation quality are provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding Fairness-Accuracy Trade-offs in Machine Learning Models: Does Promoting Fairness Undermine Performance?." pith.science (2026). https://pith.science/paper/LXJEZ7LF

@misc{pith2026241117374,
  author       = {Pith},
  title        = {Pith review of: Understanding Fairness-Accuracy Trade-offs in Machine Learning Models: Does Promoting Fairness Undermine Performance?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LXJEZ7LF}},
  note         = {Machine review of arXiv:2411.17374}
}

read the original abstract

Fairness in both Machine Learning (ML) predictions and human decision-making is essential, yet both are susceptible to different forms of bias, such as algorithmic and data-driven in ML, and cognitive or subjective in humans. In this study, we examine fairness using a real-world university admissions dataset comprising 870 applicant profiles, leveraging three ML models: XGB, Bi-LSTM, and KNN, alongside BERT embeddings for textual features. To evaluate individual fairness, we introduce a consistency metric that quantifies agreement in decisions among ML models and human experts with diverse backgrounds. Our analysis reveals that ML models surpass human evaluators in fairness consistency by margins ranging from 14.08\% to 18.79\%. Our findings highlight the potential of using ML to enhance fairness in admissions while maintaining high accuracy, advocating a hybrid approach combining human judgement and ML models.

Figures

Figures reproduced from arXiv: 2411.17374 by the authors.

**Figure 1.** Proposed pipelines to assess Individual Fairness Representations from Transformers (BERT), enabling the extraction of rich contextual representations for downstream predictive tasks. In our case, we focus on the important but challenging task of admission offer decision making. We examine individual fairness in university admission offer decisions for a heterogeneous group of applicants, characterized by diverse ba… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

[1]

In: Proc

Alur, R., Laine, L., Li, D.K., Raghavan, M., Shah, D., Shung, D.: Auditing for human expertise. In: Proc. of NeurIPS (2023)

work page 2023
[2]

Bhattacharya, D., Kanaya, S., Stevens, M.: Are university admissions academically fair? Review of Economics and Statistics99(3), 449–464 (2017)

work page 2017
[3]

Science356(6334), 183–186 (2017)

Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science356(6334), 183–186 (2017)

2017
[4]

ACM CSUR56(7), 1–38 (2024)

Caton, S., Haas, C.: Fairness in machine learning: A survey. ACM CSUR56(7), 1–38 (2024)

work page 2024
[5]

In: Proc

Dwork, C.: Differential privacy. In: Proc. of ICALP. p. 1–12 (2006)

work page 2006
[6]

In: Proc

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through aware- ness. In: Proc. of ITCS. p. 214–226 (2012)

work page 2012
[7]

ACM TiST14(5), 1–48 (2023)

Li, Y., Chen, H., Xu, S., Ge, Y., Tan, J., Liu, S., Zhang, Y.: Fairness in recommen- dation: Foundations, methods, and applications. ACM TiST14(5), 1–48 (2023)

work page 2023
[8]

In: Proc

Liu, J., Lee, R.K.W., Lim, K.H.: Bgm-han: A hierarchical attention network for accurate and fair decision assessment on semi-structured profiles. In: Proc. of ASONAM (2025)

work page 2025

Show all 18 references

[9]

O’Reilly (2020)

Mahoney, T., Varshney, K.R., Hind, M.: AI Fairness: How to Measure and Reduce Unwanted Bias in Machine Learning. O’Reilly (2020)

2020
[10]

In: Proc

Marcinkowski, F., Kieslich, K., Starke, C., Lünich, M.: Implications of ai (un-) fairness in higher education admissions: the effects of perceived ai (un-) fairness on exit, voice and organizational reputation. In: Proc. of FAT. pp. 122–130 (2020)

2020
[11]

ACM CSUR54(6), 1–35 (2021)

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM CSUR54(6), 1–35 (2021)

2021
[12]

Pitman, T.: Understanding ‘fairness’ in student selection: are there differences and does it make a difference anyway? Studies in Higher Education41(7), 1203–1216 (2016)

2016
[13]

In: Proc

Puranik, B., Madhow, U., Pedarsani, R.: A dynamic decision-making framework promoting long-term fairness. In: Proc. of AIES. pp. 547–556 (2022)

2022
[14]

Expert Systems with Applications231, 120914 (2023)

Unknown,A.:Lessonsfromdebiasingdataforfairandaccuratepredictivemodeling in education. Expert Systems with Applications231, 120914 (2023)

2023
[15]

In: Proc

Van Busum, K., Fang, S.: Analysis of ai models for student admissions: A case study. In: Proc. of SAC. pp. 17–22 (2023)

2023
[16]

ACM TOIS41(3), 1–43 (2023)

Wang, Y., Ma, W., Zhang, M., Liu, Y., Ma, S.: A survey on the fairness of recom- mender systems. ACM TOIS41(3), 1–43 (2023)

2023
[17]

In: Proc

Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representa- tions. In: Proc. of ICML. pp. 325–333 (2013)

2013
[18]

Oxford Review of Education36(3), 307–323 (2010)

Zimdars, A.: Fairness and undergraduate admission: a qualitative exploration of admissions choices at the university of oxford. Oxford Review of Education36(3), 307–323 (2010)

2010

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

In: Proc

Alur, R., Laine, L., Li, D.K., Raghavan, M., Shah, D., Shung, D.: Auditing for human expertise. In: Proc. of NeurIPS (2023)

work page 2023

[2] [2]

Bhattacharya, D., Kanaya, S., Stevens, M.: Are university admissions academically fair? Review of Economics and Statistics99(3), 449–464 (2017)

work page 2017

[3] [3]

Science356(6334), 183–186 (2017)

Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science356(6334), 183–186 (2017)

2017

[4] [4]

ACM CSUR56(7), 1–38 (2024)

Caton, S., Haas, C.: Fairness in machine learning: A survey. ACM CSUR56(7), 1–38 (2024)

work page 2024

[5] [5]

In: Proc

Dwork, C.: Differential privacy. In: Proc. of ICALP. p. 1–12 (2006)

work page 2006

[6] [6]

In: Proc

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through aware- ness. In: Proc. of ITCS. p. 214–226 (2012)

work page 2012

[7] [7]

ACM TiST14(5), 1–48 (2023)

Li, Y., Chen, H., Xu, S., Ge, Y., Tan, J., Liu, S., Zhang, Y.: Fairness in recommen- dation: Foundations, methods, and applications. ACM TiST14(5), 1–48 (2023)

work page 2023

[8] [8]

In: Proc

Liu, J., Lee, R.K.W., Lim, K.H.: Bgm-han: A hierarchical attention network for accurate and fair decision assessment on semi-structured profiles. In: Proc. of ASONAM (2025)

work page 2025