Oxford Handbook on AI Ethics Book Chapter on Race and Gender

Timnit Gebru

REVIEW 2 major objections 5 minor 19 references

Oxford Handbook on AI Ethics Book Chapter on Race and Gender

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that AI systems are not neutral: they are built by dominant groups, trained on data that encodes existing inequality, and deployed most heavily on the very people they treat worst, creating feedback loops that deepen…

desk verdict A well-sourced synthesis arguing AI reflects and amplifies social bias; the diversity remedy is plausible but asserted, not demonstrated, so it should be framed as a hypothesis. read the letter →

arxiv 1908.06165 v1 pith:DIE5BMQQ submitted 2019-08-08 cs.CY cs.AI

Timnit Gebru This is my paper · ORCID

classification cs.CYcs.AI

keywords algorithmicbiasAIethicsraceandgenderfacerecognitionfeedbackloopsintersectionalitypredictivepolicingnaturallanguageprocessing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This chapter argues that AI systems are not neutral technical tools. It claims that they are built by dominant groups, trained on data that already reflects social inequality, and deployed in settings such as policing, hiring, translation, and surveillance, where errors and stereotypes fall hardest on marginalized people. The author's core claim is that bias in AI is not an isolated defect but a feedback loop: biased tools produce outcomes that are then fed back into the system as training data, deepening the original inequality. A sympathetic reader should care because the same tools are used most heavily on the groups they serve worst, so fixing the technology requires changing who builds it and who is at the decision table, not just tweaking algorithms.

What carries the argument

The load-bearing mechanism is the runaway feedback loop: a model trained on historical data, such as hiring decisions or arrest records, learns the existing pattern of who succeeds or who is policed, its outputs are used to make real decisions, and those decisions enter the next round of training data, so small initial disparities become large structural ones. The chapter also relies on intersectional disaggregation—measuring system performance separately for combined identity categories such as darker-skinned women rather than for race or gender alone—as the method that exposes disparities a single-axis test misses.

What would settle it

Take a deployed face-recognition or hiring system, compute error rates disaggregated by skin type and gender, and compute per-capita exposure for each group (scans or screenings per person). The chapter's strongest claim predicts that the most-exposed group is also the most-misclassified group; the claim falls if the two rankings clearly diverge.

Watch

Extended reading notes

Core claim

The central claim is that AI systems mirror and amplify existing race and gender hierarchies rather than correcting them. The paper assembles evidence of this pattern across domains: commercial facial analysis has near-perfect accuracy for lighter-skinned men but error rates up to 35.5 percent for darker-skinned women; risk-assessment tools used in the criminal justice system reproduce racial disparities in arrests; word embeddings trained on news text complete the analogy “man is to computer programmer as woman is to homemaker”; and predictive policing models trained on historical arrest data create runaway feedback loops. The sharpest claim is the author's own summary: these tools are most often used on people towards whom they exhibit the most bias. The paper then argues that this is a structural problem—rooted in who creates the technology, what data it is trained on, and the unregulated high-stakes settings where it is deployed—rather than a purely technical defect that better algorithms alone can fix.

Load-bearing premise

The chapter's main remedy—diversifying who builds AI—rests on the premise that the demographic identity of technology creators determines which values are embedded in their systems; the author states this in Section 6 without empirical support, so if team diversity does not change system behavior, the proposed solution loses its foundation.

Editorial extensions

If this is right

Bias in AI should be treated as a systemic feedback problem, not a one-time model defect, because every biased decision feeds future training data and can worsen the original disparity.
Evaluating systems by overall accuracy is insufficient; intersectional subgroup evaluation becomes the minimum standard for high-stakes deployments.
Some AI applications, such as automatic gender recognition, may need to be retired rather than fixed, because the task itself encodes the harmful assumption that gender is a static binary.
Regulation and standard-setting bodies are necessary complements to technical fixes, since unregulated use in law enforcement and immigration is exactly where documented harm concentrates.
If who builds the technology determines whose values are embedded in it, then workforce representation and whose problems get funded become fairness interventions in their own right.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable implication the author leaves implicit: if the feedback-loop mechanism is right, bias audits should measure not just static accuracy gaps but how quickly deployment changes the demographic distribution of future decisions and training data.
The chapter's logic points beyond diversity headcounts to decision power and problem selection; a reasonable proxy would be measuring the share of AI research agendas and procurement rules actually set by affected communities.
The claim that tools are used most on those they bias most could be turned into an exposure-weighted bias metric: multiply per-group error rates by per-group deployment frequency, and target the systems with the highest exposure-weighted harm.
The governance argument implies that procurement rules and standardization bodies, not only algorithm tweaks, are the levers with the largest practical leverage for reducing harm.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a book chapter, not a new empirical paper. It brings together documented cases—Gender Shades, COMPAS/ProPublica, word embedding bias, the Amazon hiring tool, the Facebook Translate arrest—and reads them through a historical and political lens. The synthesis is the contribution: the parallels to parachute science, 'capture and neutralize' funding, and the runaway feedback loop in hiring and predictive policing are clearly argued and well sourced. The chapter earns its place as a strong teaching text and a serious polemic.

The factual base is solid. The cited studies are real, independently published, and on point. The author's own prior work (Gender Shades, Model Cards) appears here, but those are peer-reviewed and the chapter does not reduce to them. No circularity problem.

The soft spot is exactly where the reader and stress-test put it: Section 6's claim that 'who creates the technology determines whose values are embedded in it.' That causal claim is load-bearing for the diversity recommendation, but none of the cited cases isolate creator demographics from training data, evaluation choices, or institutional incentives. Gender Shades is about benchmark design and dataset composition; COMPAS reflects historical arrest data; word embedding bias is a corpus property; the Facebook Translate example is the closest to a demographic story, but the chapter itself also invokes data scarcity and asymmetric state power. So the mechanism is plausible but not demonstrated. This is a fixable problem: present the diversity mechanism as a hypothesis, soften the universal claims, and distinguish it from the well-supported point that the field's priorities reflect who holds funding and power.

The 'structure of the nonsense' line and the discussion of the Google Photos misclassification are the strongest parts, because they separate statistical error rates from the social meaning of specific errors.

Who should read this: anyone teaching AI ethics or wanting a one-chapter framing of the structural critique. It deserves serious refereeing; the right referee report would ask for the Section 6 reframing and a few fewer universal generalizations, not for new data. I'd take it.

Referee Report

2 major / 5 minor

Summary. This chapter argues that AI systems are not neutral: they are built by dominant social groups, trained on data that reflect existing inequalities, and deployed in ways that disproportionately harm marginalized people, creating feedback loops that deepen these harms. The author surveys documented cases – commercial facial analysis with higher error rates for darker-skinned women, the COMPAS recidivism tool's disparities, gender and racial bias in word embeddings, the Amazon hiring tool, and the Facebook Translate incident – and situates them within a broader history of 'scientific' racism and gender bias. The chapter then proposes remedies: standardization bodies, diverse teams building AI, and greater attention to historical and political context. The central diagnostic claim is well-illustrated, but the prescriptive argument in Section 6 rests on an unsupported causal assertion about who creates technology.

Significance. The chapter brings together key empirical studies and critical theory in an accessible synthesis, and it foregrounds intersectionality (via Buolamwini and Gebru's Gender Shades and Crenshaw's work) in a way that is often missing from technical fairness discussions. It also names specific institutional actors and their practices, which gives concreteness to otherwise abstract concerns. If the argument is accepted, the chapter would be a useful reference point for AI ethics courses and for policy discussions. However, the scientific contribution is limited by the unsupported inference from a small set of documented cases to AI as a whole, and by the asserted causal link between creator demographics and system values. These limitations do not invalidate the diagnosis, but they do require reframing the remedy as a hypothesis rather than a conclusion.

major comments (2)

[Section 6] The sentence 'Who creates the technology determines whose values are embedded in it' is the load-bearing step for the chapter's prescriptive conclusion. The empirical cases cited earlier do not isolate the demographic composition of the development team as the causal mechanism. Gender Shades identifies training-data and evaluation gaps; COMPAS reflects historical arrest data and court decisions; the word-embedding results stem from corpus statistics; and the Facebook Translate incident involves data scarcity for Arabic dialects and asymmetric state power. The chapter's own discussion of Amazon's 'capture and neutralize' strategy and NSF funding points to institutional incentives rather than the identity of individual engineers. To support the diversity remedy, the chapter needs evidence that changing team demographics while holding data, incentives, and institutional structures fixed changes system behavior. Absent such evidence, the claim should be presented as a hypothesis to be tested, not as a conclusion.
[Abstract and Section 6] The statement that 'these tools are most often used on people towards whom they exhibit the most bias' is a conjunction not established in the chapter. The evidence cited for disproportionate use (O'Neil 2016; Eubanks 2018) concerns the poor and marginalized being subjected to more automated decision systems, whereas the evidence for bias (Gender Shades; COMPAS; Bolukbasi et al.; Caliskan et al.) concerns specific commercial systems and their error rates or outputs. The chapter does not show that the same populations who are most often subjected to a given tool are also the populations for whom that tool's bias is largest. Without such evidence, the feedback-loop argument remains speculative, and the statement should be qualified as an inference or supported with data.

minor comments (5)

[Section 4] The statistic that '56% of the respondents who were regularly misgendered in the workplace had attempted suicide' is attributed to Hamidi et al., but that source is a secondary citation of the 2014 National Transgender Discrimination Survey; please verify the figure and clarify the provenance.
[Section 6] The term 'cis gendered' should be written as 'cisgender' or 'cis-gendered' for consistency with standard usage.
[Bibliography] The bibliography lists 'West, Sarah Myers, et al. Discriminating Systems' but this work is not cited in the text; either cite it where relevant or remove it from the bibliography.
[Footnotes] Footnote 18 is duplicated for the two citations to Buolamwini and Gebru's Gender Shades; the notes should be renumbered.
[Introduction] The chapter would benefit from an explicit statement of its scope and method, since it moves from historical analogies to contemporary AI without clarifying the intended inferential weight of those analogies.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the chapter is an argumentative synthesis using independent external evidence; its under-supported diversity-causality claim is an evidence gap, not circularity.

full rationale

This is an argumentative essay, not a formal derivation: there are no equations, no fitted parameters, and no quantity is predicted from a subset of its own inputs. The empirical anchors cited (Gender Shades error rates, the ProPublica COMPAS analysis, Bolukbasi et al. and Caliskan et al. word-embedding results, the Facebook Translate 'good morning' incident) are external published findings used as evidence for the thesis, not as outputs of the chapter. The author's own prior work appears in the citations (Gender Shades, Model Cards, and a letter by 'Concerned Researchers'), but these are peer-reviewed empirical studies or clearly narrated historical events; they are not unverified premises that the argument needs in order to force its conclusion. The strongest normative step, Section 6's claim that 'Who creates the technology determines whose values are embedded in it,' is asserted with illustrative hypotheticals rather than demonstrated, and the diversity remedy is accordingly under-evidenced. That is a support or correctness weakness, not circularity: the claim is not defined in terms of the conclusion, nor fitted from a subset of the data, nor justified solely by a self-citation chain. No step reduces by construction to its own inputs, so no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The chapter introduces no free parameters or invented entities. Its central argument rests on four assumptions: that race is a social construct, that science is never value-neutral, that who builds technology shapes its values, and that feedback loops amplify bias. The first two are standard positions in their fields, the last is supported by one cited study, and the third is a structural assumption specific to the chapter's thesis.

assumptions (4)

domain assumption Race is a social construct with no biological basis
The chapter uses this to argue that automated systems should not use race as a fixed category, citing scientists like Michael Yudell (Section 1). This is an established position in biology and anthropology, but the paper does not re-derive it.
domain assumption All science is historically and politically situated; there is no 'view from nowhere'
Section 7 invokes feminist critiques of objectivity to argue that AI research cannot be value-neutral. This is a philosophical premise, not a result proven in the chapter.
ad hoc to paper Who creates technology determines whose values are embedded in it
Section 6 states this as a causal assumption ('Who creates the technology determines whose values are embedded in it') and uses it as the basis for the proposed solution of diversifying the field. No empirical evidence for this specific causal claim is provided.
domain assumption Data-driven feedback loops in predictive systems amplify existing disparities
Section 2 asserts that models trained on biased past data create runaway feedback loops, citing Lum and Isaac's PredPol study. The chapter generalizes this mechanism to hiring and other domains without direct evidence for each.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Oxford Handbook on AI Ethics Book Chapter on Race and Gender." pith.science (2026). https://pith.science/paper/DIE5BMQQ

@misc{pith2026190806165,
  author       = {Pith},
  title        = {Pith review of: Oxford Handbook on AI Ethics Book Chapter on Race and Gender},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIE5BMQQ}},
  note         = {Machine review of arXiv:1908.06165}
}

read the original abstract

From massive face-recognition-based surveillance and machine-learning-based decision systems predicting crime recidivism rates, to the move towards automated health diagnostic systems, artificial intelligence (AI) is being used in scenarios that have serious consequences in people's lives. However, this rapid permeation of AI into society has not been accompanied by a thorough investigation of the sociopolitical issues that cause certain groups of people to be harmed rather than advantaged by it. For instance, recent studies have shown that commercial face recognition systems have much higher error rates for dark skinned women while having minimal errors on light skinned men. A 2016 ProPublica investigation uncovered that machine learning based tools that assess crime recidivism rates in the US are biased against African Americans. Other studies show that natural language processing tools trained on newspapers exhibit societal biases (e.g. finishing the analogy "Man is to computer programmer as woman is to X" by homemaker). At the same time, books such as Weapons of Math Destruction and Automated Inequality detail how people in lower socioeconomic classes in the US are subjected to more automated decision making tools than those who are in the upper class. Thus, these tools are most often used on people towards whom they exhibit the most bias. While many technical solutions have been proposed to alleviate bias in machine learning systems, we have to take a holistic and multifaceted approach. This includes standardization bodies determining what types of systems can be used in which scenarios, making sure that automated decision tools are created by people from diverse backgrounds, and understanding the historical and political factors that disadvantage certain groups who are subjected to these tools.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

Race and Gender

“Race and Gender” Timnit Gebru ABSTRACT From massive face-recognition-based surveillance and machine-learning-based decision systems predicting crime recidivism rates, to the move towards automated health diagnostic systems, artificial intelligence (AI) is being used in scenarios that have serious consequences in people's lives. However, this rapid permea...

work page 2016
[2]

USING PAST DATA TO DETERMINE FUTURE OUTCOMES RESULTS IN RUNAWAY FEEDBACK LOOPS An aptitude test designed by specific people is bound to inject their subjective biases of who is supposed to be good for the job, and eliminate diverse groups of people who do not fit the rigid, arbitrarily defined criteria that have been put in place. Those for whom the tech ...

work page 2018
[3]

Gender recognition or gender reductionism?: The social implications of embedded gender recognition systems

Hamidi, Foad, Morgan Klaus Scheuerman, and Stacy M. Branham. "Gender recognition or gender reductionism?: The social implications of embedded gender recognition systems." In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p

work page 2018
[4]

Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics

AI BASED TOOLS ARE PERPETUATING GENDER STEREOTYPES 21 Crenshaw, Kimberle. "Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics." U. Chi. Legal F. (1989): 139 12 While the previous section has discussed manners in which automated facial analysis tools with unequ...

work page 1989
[5]

By 2009, fewer than 1 26 Curtis, Cara. “Deepfakes are being weaponized to silence women—but this woman is fighting back

POWER IMBALANCE AND THE EXCLUSION OF MARGINALIZED VOICES IN AI The weaponization of technology against certain groups, as well as its usage to maintain the status quo while being touted as a liberator of those without power, is not new to AI. In Model Cards for Model Reporting, Mitchell et al. note parallels to other industries where products were designe...

work page 2018
[10]

There’s a reason Siri, Alexa and AI are imagined as female – sexism

25 Chambers, Amy. "There’s a reason Siri, Alexa and AI are imagined as female – sexism." The Conversation (2018). http://theconversation.com/theres-a-reason-siri-alexa-and-ai-are-imagined-as-female-sexism-96430 14 What does it mean for children to grow up in households filled with feminized voices that are in clearly subservient roles? AI systems are alre...

work page 2018
[13]

On Recent Research Auditing Commercial Facial Analysis Technology

17 communities negatively impacted by Amazon’s product, the company then claims to work on fairness by announcing a joint grant with NSF. This incident is a microcosm for the capture and neutralize strategy that disempowers those from marginalized communities while using the fashionable language of ethics, fairness, diversity and inclusion to advance the ...

work page 2019
[15]

attack them

22 are associated with pleasant concepts like flowers.40 Dixon et al.41 have also shown that sentiment analysis tools often classify texts pertaining to LGBTQ+ individuals as negative. Given the stereotyping of Muslims as terrorists by many western nations, it is thus less surprising to have a mistake resulting in a translation to “attack them”. This inci...

arXiv 2016

Show all 19 references

[16]

24 United States Immigration and Customs Enforcement (ICE) even more problematic and scary. The 2018 initiative proposes that ICE partners with tech companies to monitor various people’s social network data with automated tools, and use that analysis to decide whether they sho...

2018
[18]

Discriminating Systems: Gender, Race And Power in AI

West, Sarah Myers, Meredith Whittaker, and Kate Crawford. "Discriminating Systems: Gender, Race And Power in AI." AI Now Institute (2019). 47 Rogaway, Phillip. "The Moral Character of Cryptographic Work." IACR Cryptology ePrint Archive 2015 (2015):

2019
[20]

Replacing the ‘View from Nowhere’: A Pragmatist-Feminist Science Classroom

Stitzlein, Sarah M. "Replacing the ‘View from Nowhere’: A Pragmatist-Feminist Science Classroom." Electronic Journal of Science Education (2004)

2004
[1863]

Why are males over-represented at the upper extremes of intelligence?

4 Since the days of Darwin, race has been shown time and time again to be a social construct that has no biological basis.5 According to professor of public health Michael Yudell, race is “a concept we think is too crude to provide useful information, it's a concept that has s...

2013
[2010]

smartness

5 Because of the myth of scientific objectivity, these types of claims that seem to be backed up by data and “science” are less likely to be scrutinized. Just like Darwin and Hunt, many scientists today perpetuate the view that there is an inherent difference between the abili...

2018
[2012]

hotspots

8 disabilities, members of the LGBTQ+ community and any community that has been marginalized in the tech industry and in the US. The person may not be hired because of bias in the interview process, or may not succeed because of an environment that does not set up people from ...

2016
[2015]

American Journal of Sociology

The cost of color: Skin color, discrimination, and health among African-Americans. American Journal of Sociology. 121(2), pp 396-444. 11 classify images into darker and lighter skinned subjects, analyzing the accuracy of commercial systems for each of these subgroups. Buolamwi...

1976
[2017]

the view from nowhere

25 If we are to work on technology that is beneficial to all of society, it has to start from the involvement of people from many walks of life and geographic locations. The future of who technology benefits will depend on who builds it and who utilizes it. As we have seen, th...

2004
[2018]

some studies even suggest that within-race inequalities associated with skin tone among African Americans often rival or exceed what obtains between blacks and whites as a whole

10 35.5%. After this study was published, Microsoft and IBM released new versions of their APIs less than 6 months after the paper’s publication, major companies such as Google established fairness organizations, and US Senators Kamala Harris, Cory Booker and Cedric Richmond c...
[2019]

African genomes are the most diverse of any on the planet

15 percent of the several hundred genome investigations included Africans,” even though “African genomes are the most diverse of any on the planet.”28 Excluding African genes not only hurts those of African descent by creating next generation personalized drugs that do not wor...

2018
[2049]

Gender recognition or gender reductionism?: The social implications of embedded gender recognition systems

and Marvel's AIs, FRIDAY (Avengers: Infinity War), and Karen (Spider-Man: Homecoming). These names demonstrate the assumption that virtual assistants, from SatNav to Siri, will be voiced by a woman. This reinforces gender stereotypes, expectations, and assumptions about the fu...

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Race and Gender

“Race and Gender” Timnit Gebru ABSTRACT From massive face-recognition-based surveillance and machine-learning-based decision systems predicting crime recidivism rates, to the move towards automated health diagnostic systems, artificial intelligence (AI) is being used in scenarios that have serious consequences in people's lives. However, this rapid permea...

work page 2016

[2] [2]

USING PAST DATA TO DETERMINE FUTURE OUTCOMES RESULTS IN RUNAWAY FEEDBACK LOOPS An aptitude test designed by specific people is bound to inject their subjective biases of who is supposed to be good for the job, and eliminate diverse groups of people who do not fit the rigid, arbitrarily defined criteria that have been put in place. Those for whom the tech ...

work page 2018

[3] [3]

Gender recognition or gender reductionism?: The social implications of embedded gender recognition systems

Hamidi, Foad, Morgan Klaus Scheuerman, and Stacy M. Branham. "Gender recognition or gender reductionism?: The social implications of embedded gender recognition systems." In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p

work page 2018

[4] [4]

Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics

AI BASED TOOLS ARE PERPETUATING GENDER STEREOTYPES 21 Crenshaw, Kimberle. "Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics." U. Chi. Legal F. (1989): 139 12 While the previous section has discussed manners in which automated facial analysis tools with unequ...

work page 1989

[5] [5]

By 2009, fewer than 1 26 Curtis, Cara. “Deepfakes are being weaponized to silence women—but this woman is fighting back

POWER IMBALANCE AND THE EXCLUSION OF MARGINALIZED VOICES IN AI The weaponization of technology against certain groups, as well as its usage to maintain the status quo while being touted as a liberator of those without power, is not new to AI. In Model Cards for Model Reporting, Mitchell et al. note parallels to other industries where products were designe...

work page 2018

[6] [10]

There’s a reason Siri, Alexa and AI are imagined as female – sexism

25 Chambers, Amy. "There’s a reason Siri, Alexa and AI are imagined as female – sexism." The Conversation (2018). http://theconversation.com/theres-a-reason-siri-alexa-and-ai-are-imagined-as-female-sexism-96430 14 What does it mean for children to grow up in households filled with feminized voices that are in clearly subservient roles? AI systems are alre...

work page 2018

[7] [13]

On Recent Research Auditing Commercial Facial Analysis Technology

17 communities negatively impacted by Amazon’s product, the company then claims to work on fairness by announcing a joint grant with NSF. This incident is a microcosm for the capture and neutralize strategy that disempowers those from marginalized communities while using the fashionable language of ethics, fairness, diversity and inclusion to advance the ...

work page 2019

[8] [15]

attack them

22 are associated with pleasant concepts like flowers.40 Dixon et al.41 have also shown that sentiment analysis tools often classify texts pertaining to LGBTQ+ individuals as negative. Given the stereotyping of Muslims as terrorists by many western nations, it is thus less surprising to have a mistake resulting in a translation to “attack them”. This inci...

arXiv 2016