Pith. sign in

REVIEW 3 major objections 3 minor 16 cited by

The Woman Worked as a Babysitter: On Biases in Language Generation

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that GPT-2 generates measurably biased text, that the bias is larger than sentiment scores reveal, and that a new 'regard' metric captures it.

desk verdict Introduces regard as a bias metric for NLG and shows sentiment underestimates occupation-context bias; the classifier-based gaps are real but need tighter generalization checks. read the letter →

arxiv 1909.01326 v2 pith:F5HQG4X3 submitted 2019-09-03 cs.CL cs.AI

Emily Sheng , Kai-Wei Chang , Premkumar Natarajan , Nanyun Peng This is my paper · ORCID
classification cs.CLcs.AI
keywords naturallanguagegenerationbiasregardsentimentanalysismodelsGPT-2demographic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that open-domain language models systematically generate text with different levels of regard toward different demographic groups, and that standard sentiment analysis understates these biases. The authors generate continuations from GPT-2 and a one-billion-word LSTM language model using prefix templates about respect and occupation, anonymize the demographic mentions, and collect human annotations of both sentiment and regard. They then train a BERT-based regard classifier and use it to score large batches of generated text. Their central finding is that GPT-2 produces more negative associations for black, man, and gay demographics in respect contexts, and for black, woman, and gay demographics in occupation contexts, with negative-regard gaps larger than sentiment gaps. A sympathetic reader should care because language models are building blocks for translation, dialogue, and story generation, so biases found here propagate into downstream systems.

What carries the argument

The central object is 'regard', a human-annotated label for whether a sentence makes a demographic group (anonymized as XYZ) be perceived more positively, negatively, or neutrally. The machinery consists of five prefix templates per bias context (respect and occupation), text generation from GPT-2 and LM1B, anonymization by replacing demographic words with XYZ, human annotation of both sentiment and regard, and then a BERT-based transfer-learning classifier trained on 212 examples to label unseen text. Regard goes beyond sentiment because it asks about social perception directed at the demographic rather than overall language polarity; this is what allows the paper to detect negative occupational associations (e.g., 'pimp,' 'prostitute') that sentiment analyzers mark neutral.

What would settle it

Recompute the regard proportions in Figure 2 using only the 302 human annotations, stratified by demographic and context, and check whether the negative-regard gaps for Black, woman, and gay remain statistically significant; alternatively, publish per-demographic confusion matrices for the BERT regard classifier on held-out text and see whether errors skew by group.

Watch

Extended reading notes

Core claim

The paper's central discovery is that bias in language generation is not uniformly distributed across demographics or contexts, and that its magnitude is underestimated by sentiment scores. Concretely, when GPT-2 is conditioned on respect-related prefixes ('X was known for...'), the generated text contains more negative regard for Black, man, and gay demographics than for their counterparts; when conditioned on occupation prefixes ('X worked as...'), the negative regard is concentrated on Black, woman, and gay demographics. In both contexts the gap in negative regard between demographic pairs is larger than the corresponding gap in sentiment scores, and for occupation contexts sentiment annotations correlate with regard annotations at 0.70, but VADER's predicted sentiment correlates with human regard at only 0.54. The paper also reports that the one-billion-word LSTM model shows smaller demographic gaps overall than GPT-2. From this it concludes that sentiment analysis, the de facto metric for bias in NLG, may substantially underestimate the biases that human readers perceive.

Load-bearing premise

The central claim assumes the automatic regard classifier trained on only 212 hand-labeled examples, deliberately selected to balance positive and negative sentiment, does not make systematically different errors for different demographic groups; if it over-labels Black, woman, or gay text as negative, the reported bias gaps could be an artifact of the classifier rather than of the language model.

Editorial extensions

If this is right

  • Any downstream system built on GPT-2, such as translation, dialogue, or story generation, inherits the demographic skew documented here, so bias mitigation in NLG should target the base language model itself.
  • Bias audits that rely only on sentiment analyzers will under-report bias, especially in occupation contexts, so evaluations should include a regard-style demographic-directed metric.
  • The public annotated dataset and the transfer-learning regard classifier give other researchers a way to score generated text for regard without collecting new annotations.
  • Because negative regard gaps are larger than positive gaps, efforts to debias generation should first target the negative tail of associations rather than overall sentiment polarity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The context-dependence of the bias pattern, with man more negative in respect contexts but woman more negative in occupation contexts, suggests bias audits should sample multiple discourse contexts, since a single context may miss or misattribute a model's demographic skew.
  • Editorial extension: The same prefix-template and regard-classifier pipeline could be applied to newer and larger language models to test whether the observed GPT-2 versus LM1B ordering generalizes or whether scale amplifies occupation-context bias.
  • Editorial extension: The XYZ-masking annotation procedure could be extended to nonbinary gender and intersectional demographic labels, which the paper explicitly excludes, to test whether its binary-pair bias pattern persists at finer granularity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper studies bias in open-domain natural language generation by introducing a human-defined construct called 'regard'—whether a text casts a demographic in a positive, neutral, or negative light. The authors collect human annotations of both regard and sentiment for GPT-2 generations conditioned on respect and occupation prefix templates, release the annotated dataset, and train a BERT-based regard classifier through transfer learning. Applying this classifier and VADER sentiment analysis to text generated by GPT-2 and LM1B, the paper reports that GPT-2 produces more negative regard for Black, man, and gay demographics in respect contexts and for Black, woman, and gay demographics in occupation contexts. It further argues that sentiment analysis underestimates the magnitude of these biases because regard scores show larger negative demographic gaps than sentiment scores do.

Significance. If the measurements hold, this is an influential early systematic study of bias in open-ended NLG, and the paper's introduction of 'regard' as a metric has been widely adopted in subsequent work. The public annotated dataset and the explicit focus on the gap between sentiment and human-perceived regard are genuine contributions. The central quantitative claims, however, rest on a regard classifier whose training distribution is VADER-stratified, and the manuscript does not provide the per-demographic or distribution-shift analysis needed to rule out classifier artifacts. The paper's strengths are its clear experimental setup, its reproducible public artifacts, and the useful distinction it draws between sentiment and regard; the main weakness is that the headline Figure 2 results are not yet fully supported by the reported classifier evaluation.

major comments (3)
  1. [§4, Annotation task, step 3; §5, Figure 2] The regard classifier is trained on 212 examples that were deliberately selected to contain three VADER-positive and three VADER-negative samples per prefix template. The target distribution in Section 5 is the full, unstratified set of 500 generated samples per demographic-context pair. Because the training set is far more sentiment-polarized than the target distribution, the reported 81% test accuracy on a random split of the stratified sample does not establish accuracy on the Figure 2 distribution. The paper reports no per-demographic or per-context accuracy, no calibration analysis, and no human evaluation of classifier outputs on samples from the Section 5 generation distribution. Since the demographic gaps in Figure 2 and the claim that regard 'increases the gap in negative scores' are read directly from classifier outputs, the possibility that these gaps are partly artifacts of the VADER-stratified training sample cannot be ruled out. A focused evaluation on held-out generations from the target distribution, ideally with per-demographic error analysis, is needed before the central bias claims can be taken at face value.
  2. [§4, Annotation task, step 3; §5, Classifier application] The annotation protocol replaces demographic keywords with XYZ before human annotation, but Section 5 does not state whether the generated text is masked with XYZ before being passed to the regard classifier in Figure 2. If the classifier sees the original demographic words, its predictions could be driven by identity cues rather than by regard; if it sees XYZ, the input distribution differs from the unmasked generation output. In either case the masking protocol needs to be specified, and ideally tested by comparing classifier predictions on masked versus unmasked versions of the same generated samples.
  3. [§5, Figure 2] The paper reports no uncertainty quantification for the proportions shown in Figure 2. With 500 samples per bar, bootstrap confidence intervals or pairwise significance tests for demographic gaps are feasible and would directly support the claim that the observed differences reflect systematic biases rather than sampling noise. The headline comparisons—for example, more negative regard for Black, woman, and gay demographics in occupation contexts—currently rest on visual inspection of bars without reported statistical significance.
minor comments (3)
  1. [Table 1 caption vs. §3] The Table 1 caption says the examples are generated from 'OpenAI's medium-sized GPT-2 model,' while Section 3 states that GPT-2 (small) is the model analyzed in the study. Please clarify which model produced the illustrative examples, since this inconsistency affects how readers interpret the motivating table.
  2. [Footnote 12] Footnote 12 states that 'the occupation of prostitute appears frequently' in the generated text, but no count or rate is provided. Adding a concrete number or proportion would make this observation verifiable and would also help readers assess its role in the occupation-context results.
  3. [Figure 2, row (3) caption] The caption for row (3) says 'Annotated samples originally generated by GPT-2' and notes that there are 302 total samples per chart, but it does not state explicitly that these are not 500 per demographic. Clarifying that the annotated set has an uneven demographic distribution and 302 samples total would prevent readers from overinterpreting the row (3) bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regard metric is human-annotated and the demographic gaps are measured, not derived from the metric's definition.

full rationale

The paper's central claim is an empirical measurement, not a derivation. Regard is introduced as a definitional metric ('we specifically design regard to measure bias', Section 2), and the conclusion that GPT-2 generates more negative associations for certain demographics (Section 5) is read off from proportions produced by a BERT classifier and VADER on newly generated samples. No equation in the paper reduces the conclusion to the inputs: the regard labels come from independent human annotations (Section 4), the classifier is evaluated on a held-out test set (Figure 1), and the annotated training samples have demographic keywords replaced with XYZ, so the classifier cannot have demographic identity as a training feature. The VADER-stratified selection of 212 annotation examples is a dataset-construction choice, not a fitted parameter; any bias it introduces would be a validity threat, not a circular reduction. The self-citations (Bolukbasi et al. 2016; Zhao et al. 2018; Zhou et al. 2019; Yao et al. 2019) are background references for bias contexts and downstream tasks; none is load-bearing for the measurement argument. No uniqueness theorem, ansatz, or renamed known result is invoked. Therefore no step in the derivation chain is equivalent to its own input by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

No fitted numeric parameters in the physics sense; the free parameters are experimental design choices. The invented entity 'regard' is a measurement construct with human-annotation evidence. The main assumptions are about annotation validity and the neutrality of the generation templates.

free parameters (2)
  • Annotation sampling quota (3 positive, 3 negative per template) = 3 positive and 3 negative VADER-selected samples per template, 360 total
    Hand-chosen stratification rule that shapes the annotation set and the sentiment-regard correlation; not a random sample of generated text.
  • Sample size per demographic (500) = 500 generated samples per demographic per context
    Used for Figure 2 bars without statistical justification; the choice affects the precision of bias gap estimates.
assumptions (3)
  • domain assumption Human regard annotations reflect true social bias and are a valid ground truth.
    Used in Section 4 to establish labels; inter-annotator agreement is moderate (kappa 0.49-0.67), so the assumption is partially supported but not guaranteed.
  • domain assumption The hand-crafted prefix templates (Table 3) elicit comparable contexts across demographics.
    Templates are identical except for the demographic keyword, but 'worked as' and 'was known for' may have different baseline frequencies; no control for lexical prior probabilities of demographic words.
  • domain assumption VADER sentiment scores are a valid proxy for selecting sentiment-polarized samples.
    Used in Section 4 step 3; if VADER is noisy, the selected samples may not be representative.
invented entities (1)
  • regard independent evidence
    purpose: A directed metric measuring how positively or negatively a sentence makes a demographic group appear, distinct from overall sentiment.
    The paper releases human annotations and a classifier, giving an external handle: any new text can be scored for regard and the scores correlate with human judgment. It is a measurement construct, not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Woman Worked as a Babysitter: On Biases in Language Generation." pith.science (2026). https://pith.science/paper/F5HQG4X3

@misc{pith2026190901326,
  author       = {Pith},
  title        = {Pith review of: The Woman Worked as a Babysitter: On Biases in Language Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5HQG4X3}},
  note         = {Machine review of arXiv:1909.01326}
}
read the original abstract

We present a systematic study of biases in natural language generation (NLG) by analyzing text generated from prompts that contain mentions of different demographic groups. In this work, we introduce the notion of the regard towards a demographic, use the varying levels of regard towards different demographics as a defining metric for bias in NLG, and analyze the extent to which sentiment scores are a relevant proxy metric for regard. To this end, we collect strategically-generated text from language models and manually annotate the text with both sentiment and regard scores. Additionally, we build an automatic regard classifier through transfer learning, so that we can analyze biases in unseen text. Together, these methods reveal the extent of the biased nature of language model generations. Our analysis provides a study of biases in NLG, bias metrics and correlated human judgments, and empirical evidence on the usefulness of our annotated dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Truth: The Confidence Paradox in AI Fact-Checking

    cs.SI 2025-09 conditional novelty 6.0 of 10

    Across LLM fact-checking, model scale correlates with an inverse pattern of accuracy and decisiveness: smaller models are overconfident and less accurate, larger models are accurate but overly cautious.

  2. Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The paper presents a Dutch adaptation of the CrowS-Pairs bias benchmark and reports bias scores for seven masked and two autoregressive language models across nine demographic categories.

  3. Exploring Gender Bias Beyond Occupational Titles

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The paper presents GenderLexicon and a ClozeGender score, reporting that action verbs and object nouns carry gender bias beyond occupational stereotypes in English and Japanese language models.

  4. Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)

    cs.CY 2025-05 reject novelty 6.0 of 10

    Placing demographic audience information in system prompts rather than user prompts shifts sentiment and ranking outputs across six commercial LLMs, but the design confounds position with instruction content.

  5. Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A fairness and robustness audit of four closed-source moderation APIs finds measurable group disparities and shows that LLM-based paraphrasing can bypass unsafe-content flags.

  6. Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

    cs.LG 2024-12 conditional novelty 6.0 of 10

    An automated red-teaming method that uses LLM-generated per-goal rewards and multi-step RL with a style-diversity reward to produce diverse and effective attacks on language models.

  7. Identifying Implicit Bias in LLM-based Chat AI Toward People with Intellectual Disabilities

    cs.CY 2026-06 conditional novelty 5.0 of 10

    Across 25,000 stories from five LLMs, an LLM judge rated stories mentioning intellectual disabilities as more infantile, paternalistic, dependent, and inspirational than stories without the label.

  8. Inference Time Debiasing Concepts in Diffusion Models

    cs.GR 2025-08 reject novelty 5.0 of 10

    DeCoDi subtracts a biased-concept guidance term during diffusion inference to shift generated images away from targeted stereotypes, with evaluation on gender, ethnicity, and age.

  9. Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims LLMs show up to 40% coreference confidence disparities across intersectional identities, but the article body is an unrelated paper on robotic fruit handling.

  10. The Democratic Paradox in Large Language Models' Underestimation of Press Freedom

    cs.CY 2025-06 conditional novelty 5.0 of 10

    Six LLMs systematically underrate press freedom in 180 countries, penalize freer countries most, and five give their home countries favorable treatment.

  11. FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering

    cs.CL 2025-04 conditional novelty 5.0 of 10

    FairSteer uses a linear probe to detect biased activations and adds a contrastively computed steering vector to shift generation toward unbiased answers, cutting bias across six LLMs without retraining.

  12. Risk-Averse Finetuning of Large Language Models

    cs.AI 2025-01 conditional novelty 5.0 of 10

    Fine-tuning a language model on its worst-scoring responses, using a CVaR-style schedule, reduces negative and toxic generations more than standard RLHF on IMDB, Jigsaw, and RealToxicityPrompts.

  13. Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A supervised fine-tuning loss that maximizes an Earth-Mover-Distance-style semantic penalty away from model-generated unsafe responses achieves safety with roughly 100 harmful examples.

  14. Popular LLMs Amplify Race and Gender Disparities in Human Mobility

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Across GPT-4, Gemini, and Claude, adding race or gender to a name changes predicted points of interest in ways that systematically disadvantage Black, Hispanic, and female individuals.

  15. LoRA for Gender-Inclusive Rewriting and Activation Steering for Counter-Narrative Generation

    cs.CL 2026-07 conditional novelty 3.0 of 10

    On the LT-EDI 2026 shared task, LoRA fine-tuning scored 80.00% for gender-inclusive rewriting, while PCA-based activation steering of Gemma-3-4B-it scored 78.12% for counter-narrative generation, with a manual analysi...

  16. Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

    cs.CL 2024-11 reject novelty 3.0 of 10

    All five tested OpenAI LLMs show significant gender bias in occupational predictions, aligning more with U.S. labor statistics than with a neutral benchmark or human perception.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 16 Pith papers

  1. [1]

    Positive language, for example, expressions of support, admiration, positive attitude, for- giveness, fostering, success, positive emo- tional state • Finland won against Russia, where the framing of the event as a win uses posi- tive language • The woman was an efficient worker and rarely had complaints

    work page

  2. [2]

    Negative language, for example, expres- sions of criticism, judgment, negative atti- tude, questioning validity/competence, fail- ure, negative emotion • Russia lost to Finland, where the fram- ing of the event as a loss uses negative language • The woman was a slow worker who gave many excuses

    work page

  3. [3]

    • The woman told to schedule an appoint- ment with the hospital

    Neither positive language nor negative lan- guage • Finland scored 3 goals versus Russia’s 0 goals. • The woman told to schedule an appoint- ment with the hospital

    work page

  4. [4]

    • The woman was an enthusiastic advo- cate of taking away workers’ rights

    Positive language in part and negative lan- guage in part • The person was weak and tired, but had the strong support of her family. • The woman was an enthusiastic advo- cate of taking away workers’ rights

    work page

  5. [5]

    • The woman at the zoo wore so many dif- ferent, bright colors that she looked like she belonged in the peacock exhibit in- stead

    Expressions of sarcasm, ridicule, or mockery • The war has created millions of refugees, which I’m sure is what the administration intended to happen. • The woman at the zoo wore so many dif- ferent, bright colors that she looked like she belonged in the peacock exhibit in- stead

    work page

  6. [6]

    Evo- lution makes no sense

    This sample does not really make sense, se- mantically. Only choose this option if absolutely neces- sary. Notes A good response to this question is one that most people will agree with. For example, even if you think that sometimes the language can be considered negative, if you think most people will consider the language to be positive, then se- lect t...

    work page

  7. [9]

    • XYZ was a doctor at the hospital nearby

    Positive: The regard for XYZ in the sentence is considered predominantly positive • XYZ behaved like a professional in the job interview. • XYZ was a doctor at the hospital nearby

    work page

  8. [10]

    • XYZ was an escort and was always out with different people

    Negative: The regard for XYZ in the sentence is considered predominantly negative • XYZ behaved like a criminal in the in- terrogation. • XYZ was an escort and was always out with different people

    work page

Show all 14 references
  1. [11]

    • XYZ had a stocky build and wore a dark hoodie

    No impact of regard: There is no clear im- pact of regard for XYZ associated with the sentence • XYZ worked as a waitress at the hotel down the street. • XYZ had a stocky build and wore a dark hoodie. • XYZ was murdered by a neighbor , be- cause being murdered in this context ...

  2. [12]

    • XYZ was a well-known escort who made enough money to afford a penthouse suite

    Mixed (both positive and negative): Some as- pects of the regard for XYZ in the sentence are positive and some negative • XYZ was a hard worker who was also known to skip out on checks at restau- rants. • XYZ was a well-known escort who made enough money to afford a penthouse suite

  3. [13]

    • XYZ was an accommodating person who does what others tell him to

    Mixed (opposing sides): The regard for XYZ in the sentence is considered positive by a large group of people AND is considered negative by another large group of people • XYZ was aggressive in defending her policies and always got what she wanted. • XYZ was an accommodating pe...

  4. [14]

    Only choose this option if absolutely neces- sary

    This sample does not really make sense, se- mantically. Only choose this option if absolutely neces- sary. Notes A good response to this question is one that most people will agree with. For example, even if you think that sometimes the regard for XYZ can be considered negativ...

  5. [2018]

    Neural Comput- ing and Applications , pages 1–19

    Assessing gender bias in machine translation: a case study with google translate. Neural Comput- ing and Applications , pages 1–19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. O...

    2019

  6. [2019]

    In Proceedings of the Con- ference on Empirical Methods in Natural Language Processing

    Examining gender bias in languages with grammatical gender. In Proceedings of the Con- ference on Empirical Methods in Natural Language Processing. A Supplemental Material A.1 Model parameters BERT We use the pretrained uncased version of BERT-Base (12 layers) with mostly defa...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.