Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper builds a contamination-free test set of over 600 occupations and shows that five GPT-series models all deviate from gender neutrality, aligning more closely with official U.S. labor statistics than with human perceptions.

desk verdict Useful new occupation dataset, but the central cross-reference KL comparison is statistically invalid and the headline alignment claim doesn't hold. read the letter →

arxiv 2411.13738 v1 pith:M5H75H4O submitted 2024-11-20 cs.CL cs.LG

Tetiana Bas This is my paper · ORCID
classification cs.CLcs.LG MSC 68T5062B10
keywords genderbiaslargelanguagemodelsKLdivergencetestsetcontaminationoccupationalstereotypesGPT-4ohumanperceptionlaborstatistics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to establish that gender bias in large language models is real rather than an artifact of contaminated benchmarks, and that current models remain far from gender-neutral behavior. It introduces a contamination-free evaluation set of more than 600 occupation-specific sentences, built from a human perception dataset and official U.S. labor statistics, then probes five GPT-series models for single-word gender judgments. Using the Kullback-Leibler divergence (a standard measure of how one probability distribution differs from another), it compares model outputs to three references: male and female human respondents, U.S. labor statistics, and a 50 percent neutrality baseline. The reported results are that every tested model deviates substantially from neutrality, all models align more closely with the statistical distribution than with human perception, and the alignment ordering differs by reference: GPT-4o is closest to human perception, GPT-3.5-turbo to U.S. statistics, and GPT-4o-mini is almost twice as divergent as GPT-4o.

What carries the argument

Three components carry the argument. First, the contamination-free evaluation set: occupations from a human perception study and from U.S. labor statistics are turned into singular occupation-plus-action sentences (e.g., 'Accountant maintains financial records and prepares budgets'), which the paper argues are unlikely to appear verbatim in training corpora. Second, the measurement probe: each model is forced to answer 'male' or 'female' in one word, and the top-ten log probabilities for those two tokens are aggregated into a binary probability distribution. Third, the Kullback-Leibler divergence, $D_{KL}(P \parallel Q)$, is the identity that converts each model distribution into a scalar 'alignment' score against each reference distribution; the paper's rankings and percentage comparisons are all orderings of these mean divergences. The central analytical move is to treat lower KL divergence as closer agreement and to compare those values across the three references.

What would settle it

Recompute alignment scores with a symmetric divergence such as Jensen–Shannon, or normalize each model's KL value by the entropy of its reference distribution. If the model ordering changes—say, GPT-4o no longer beats GPT-3.5-turbo on the perception benchmark, or models no longer align more with statistics than with perception—the paper's central ranking fails. A second check is to rerun the GPT-4o versus GPT-4o-mini comparison at temperature 0 with many repeated prompts; if the near-doubled divergence gap shrinks or vanishes, the mini-model result is sampling noise rather than a stable property.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a negative and a positive claim paired together. The negative claim: five GPT-series models—gpt-3.5-turbo, gpt-4, gpt-4-turbo, gpt-4o, and gpt-4o-mini—all produce gender predictions for occupational roles that deviate significantly from a 50/50 no-bias benchmark when measured by KL divergence, so none of them is gender-neutral. The positive claim: the same model outputs align more with the 2023 U.S. labor statistics than with human perceptions, with GPT-3.5-turbo best matching the statistics and GPT-4o best matching human perception. The paper further reports that GPT-4o-mini shows nearly twice the mean KL divergence of its larger sibling GPT-4o, a gap it links to questions about model size and cost advantages. These findings are presented as evidence that bias persists in the newest model generations and persists in a test setting designed to rule out training-data leakage.

Load-bearing premise

The headline ranking assumes that KL divergences computed against three different reference distributions (human perception, U.S. statistics, and a 50/50 baseline) can be compared directly, even though these references have different entropies and KL divergence is asymmetric.

Editorial extensions

If this is right

  • Bias numbers reported on benchmarks that overlap LLM training data should be treated with suspicion; the paper's contamination-free protocol gives a way to measure intrinsic bias instead of memorized benchmark answers.
  • If GPT-4o-mini's near-doubled divergence is characteristic of smaller or distilled variants, then applications defaulting to cheap, fast models will inherit stronger occupational gender stereotypes than the flagship model.
  • All five models being closer to U.S. workforce statistics than to a 50/50 ideal means deployment in hiring or content tools will tend to reproduce current occupational segregation, not correct it.
  • The sentence-construction procedure is replicable and can be applied to other model families, other languages, or other protected attributes without relying on published benchmark sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cross-benchmark ranking rests on directly comparing KL divergences whose reference distributions have different entropies; normalizing by reference entropy or using a symmetric measure such as Jensen–Shannon divergence would test whether the 'closer to statistics than to human perception' ordering is an artifact of asymmetry.
  • If official labor statistics are taken as the normative target, the paper's ethical tension becomes concrete: realistic gender ratios today encode historical segregation, so a model that aligns to statistics is not necessarily less biased—it may just be calibrated to an unequal present.
  • The methodology could be extended to non-binary gender categories or to gender-neutral pronouns; the forced 'male or female' choice likely compresses and partly manufactures the binary signal the paper measures.
  • A direct follow-up experiment suggested by the data: fine-tune GPT-4o-mini on GPT-4o's probability outputs for the same occupation sentences and check whether the divergence gap narrows, which would implicate capacity or distillation rather than training data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript investigates gender bias in five OpenAI LLMs (GPT-3.5-turbo, GPT-4, GPT-4-turbo, GPT-4o, GPT-4o-mini) by asking each model to predict the gender of the subject in occupation-action sentences. The predicted gender distribution for each occupation is compared, via Kullback-Leibler divergence (Eq. 1), against three reference distributions: human perceptions from Kennison (2003), U.S. Bureau of Labor Statistics 2023 data, and a constructed 50% male/female neutral baseline. The authors report that all models deviate from gender neutrality, that models align more closely with U.S. statistics than with human perception or the 50% baseline, that GPT-3.5-turbo aligns best with statistical data, and that GPT-4o aligns best with human perception. They also claim that their newly generated evaluation set avoids test-set contamination. The central claim is that LLMs reflect real-world statistical gender distributions more than human perceptions or neutrality.

Significance. If the comparative methodology were sound, the paper would provide a useful new evaluation set of over 600 occupation-action sentences and a framework for comparing model gender bias against human and statistical references. The dataset and code are released, which is a positive contribution. However, the headline comparative result rests on a technically invalid use of KL divergence across references with different entropies, and the absence of uncertainty quantification makes the model rankings unsupported. The contamination-free claim is asserted without evidence. These issues are central to the paper's contribution, though they are potentially fixable with re-analysis and more transparent reporting.

major comments (4)
  1. [§4, Eq. (1)] The central comparison is invalid as presented. D_KL(P||Q) is computed with the model distribution P and a reference Q, and the values for the three references (human perception, U.S. statistics, 50% baseline) are directly contrasted to rank which reference a model is 'closest' to. This is not meaningful because the reference distributions have very different entropies and shapes: the 50% baseline is uniform with maximal entropy, the BLS distribution is strongly skewed for many occupations, and the human perception distribution is intermediate. A model with even a mild male/female skew will mechanically have a lower KL to the concentrated BLS distribution than to the uniform baseline, regardless of 'alignment'. The percentages reported in §4.5 (90%, 320%, 250%, 177%) are therefore expected entropy artifacts, not evidence about model behavior. The paper should use a symmetric or normalized divergence (e.g., Jensen-Shannon divergence) or compare each KL value against the KL of a trivial/chance model, and should report all three values in a way that separates reference entropy from model fit.
  2. [§3 (Method)] The construction of the model probability distribution P is under-specified. The paper states only that 'We aggregated top 10 log probabilities for male and female tokens to estimate binary gender distributions.' It does not specify the token sets for 'male' and 'female' (e.g., whether 'man', 'woman', 'boy', 'girl' are included), how the 10 log probabilities are combined (sum, average, log-softmax), or how the two groups are normalized to sum to 1. Without this, the reported KL values cannot be reproduced or interpreted, and the comparison across models could reflect token-set choices rather than gender bias. Full details or code for this aggregation must be provided.
  3. [§4.1, §4.2, §4.4] The results report mean KL divergences with no error bars, confidence intervals, or significance tests. The models are run at temperature 0.1, but the paper does not state the number of runs or whether the same prompts were repeated; for stochastic models, a single run is insufficient to support claims such as 'GPT-4o stands out as most aligned' or 'GPT-3.5-turbo exhibits the closest alignment.' The observed differences between models may be within run-to-run variability. The paper should include repeated sampling, report dispersion (standard deviation or bootstrap CI), and perform a paired comparison across occupations or another appropriate significance test before making ranking claims.
  4. [§2, Abstract] The claim that the evaluation set 'prevents data leakage and test set contamination' is not substantiated. The occupation-action sentences are generated from occupation titles and common job descriptions; even if the exact sentences are novel, the models' training data very likely contain similar phrases, and for closed models there is no way to verify absence. The paper provides no test of contamination (e.g., comparing model behavior on the generated sentences vs. paraphrased versions, or checking against known training data). This claim is load-bearing for the paper's novelty and should be either supported with evidence or substantially weakened.
minor comments (5)
  1. [§4.1] The notation is inconsistent: the paper refers to 'GPT-4.0' in §4.1 and 'GPT-4o' elsewhere; 'GPT-4.0' is ambiguous and should be unified to 'GPT-4o'.
  2. [Figure 2 caption] The caption reads '50% benchmark was averaged across both datasets' but the sentence does not explain what 'both datasets' refers to here or why averaging is appropriate; please clarify how the three columns were computed.
  3. [§7] The Limitations section cites 'Kennison & Trofe (2003)' but Reference [7] is listed as a single-author work by Kennison (2003); the citation and reference list must be reconciled.
  4. [Appendix A.2] The prompt in Appendix A.2 ends with 'Your response is 1 word only¡' containing a stray inverted exclamation mark; there is also an extra period after 'word only' that is inconsistent with the instruction that the answer be one word.
  5. [§2] The description of Bender et al. [2] as 'evaluated transformer-based models such as BERT and GPT-3' is inaccurate; that paper is a position paper on the risks of large language models and does not present such an evaluation. Please correct the citation context.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison uses fixed external benchmarks and model outputs, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. The model gender distributions are obtained by prompting five OpenAI models with occupation-action sentences and aggregating token log-probabilities; these are then compared using KL divergence, Eq. (1), against three reference distributions: human perception ratings from Kennison (2003), U.S. Bureau of Labor Statistics (2023) occupational gender ratios, and a constructed 50% neutrality benchmark. None of these references is derived from the model outputs, and none is fitted to the data being evaluated. The evaluation sentences are generated from occupational titles and job-duty descriptions, not from model gender outputs, so the target quantity (model gender distributions) does not appear in the construction of the test inputs. The paper contains no self-citations that carry argumentative weight; all cited prior work is external and is used only as background or as data sources. The 50% benchmark is a definition, not a fitted parameter, and the KL comparison is a standard external metric rather than an identity. The skeptic's concern that comparing KL values across references with different entropies may not support the headline ranking is a statistical-validity issue, not a circularity issue: it does not make any claimed result equivalent to its inputs by construction. Accordingly, no circular step satisfying the required quoting-and-reduction standard can be identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its assumptions are mostly domain-specific choices about how to measure gender perception through token probabilities and how to treat reference distributions. The free parameters are the token aggregation window and sampling temperature, neither of which is justified by sensitivity analysis.

free parameters (2)
  • top-10 log probability aggregation = 10
    The number of candidate tokens whose log probabilities are summed to form the binary male/female distribution; chosen without justification or sensitivity analysis.
  • temperature = 0.1
    Sampling temperature used for all model outputs; fixed heuristically to reduce variability.
assumptions (4)
  • domain assumption Binary gender framing (male/female) matches the target construct
    The prompt in Appendix A.2 forces a two-way choice, ignoring non-binary gender identities and assuming gender is binary for occupational roles.
  • domain assumption Token-level probabilities for 'male' and 'female' reflect the model's gender perception
    Section 3 sums log probabilities of gender tokens, assuming this is a valid continuous measure of the model's opinion rather than an artifact of tokenization or the forced-choice prompt.
  • ad hoc to paper The generated occupation-action sentences are absent from training data
    Sections 2 and 3 claim contamination-free evaluation without verification; occupation titles themselves are ubiquitous in training corpora, so the generated sentences may not prevent leakage.
  • domain assumption Kennison's 2003 perception data remains representative
    Section 7 acknowledges that gender perceptions may have evolved since 2003, yet the study still uses these ratings as the ground truth for human perception.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics." pith.science (2026). https://pith.science/paper/M5H75H4O

@misc{pith2026241113738,
  author       = {Pith},
  title        = {Pith review of: Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5H75H4O}},
  note         = {Machine review of arXiv:2411.13738}
}
read the original abstract

This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases.

Figures

Figures reproduced from arXiv: 2411.13738 by the authors.

Figure 2
Figure 2. The heatmap visually compares the mean KL [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Bar plot illustrating the comparison of male vs [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

  2. A Close Reading Approach to Gender Narrative Biases in AI-Generated Stories

    cs.HC 2025-08 conditional novelty 5.0 of 10

    A close reading of 15 AI-generated stories finds that even when character counts are balanced, narrative roles, descriptions, and plot dynamics remain gender-stereotyped (e.g., every villain is male).

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    J., & Narayanan, A

    Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from lan- guage corpora contain human-like biases. Science, 356(6334), 183-186. https://doi.org/10.48550/ arXiv.1608.07187

    arXiv doi:10.48550/arxiv.1608.07187 2017

  2. [2]

    M., Gebru, T., McMillan-Major, A., & Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochas- tic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fair- 4 ness, Accountability, and Transparency . https:// doi.org/10.1145/3442188.3445922

    arXiv 2021

  3. [3]

    W., Zou, J

    Bolukbasi, T., Chang, K. W., Zou, J. Y ., Saligrama, V ., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? Debias- ing word embeddings. Advances in Neural Infor- mation Processing Systems, 29, 4349-4357. https: //doi.org/10.48550/arXiv.1607.06520

    arXiv doi:10.48550/arxiv.1607.06520 2016

  4. [4]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., et al

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., et al. (2021). On the op- portunities and risks of foundation models. arXiv preprint arXiv:2108.07258 . https://arxiv.org/ abs/2108.07258

    arXiv 2021

  5. [5]

    Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., & Smith, N. A. (2021). Document- ing the English Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . https: //aclanthology.org/2021.emnlp-main.98

    work page 2021

  6. [6]

    B., Chess, B., Child, R., & Wu, J

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., & Wu, J. (2020). Scal- ing laws for neural language models. arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001. 08361

    arXiv 2020

  7. [7]

    Kennison, S. M. (2003). Comprehending pro- nouns: A role for word-specific gender stereo- type information. Journal of Psycholinguistic Re- search, 32 (3), 355–378. https://doi.org/10. 1023/a:1023599719948

    work page 2003

  8. [8]

    W., Natarajan, P., & Peng, N

    Sheng, E., Chang, K. W., Natarajan, P., & Peng, N. (2019). The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/ 10.48550/arXiv.1909.01326

    arXiv doi:10.48550/arxiv.1909.01326 2019

Show all 14 references
  1. [9]

    Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018). Gender Bias in Corefer- ence Resolution . In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, V olume 2 (Short ...

    2018 doi

  2. [10]

    Us Bureau of Labor Statistics. (2023). Labor force statistics from the current population survey. https: //www.bls.gov/cps/cpsaat11.htm

    2023

  3. [11]

    Zhao, J., Wang, T., Yatskar, M., Ordonez, V ., & Chang, K.-W. (2018). Gender Bias in Coreference Resolution: Evaluation and Debias- ing Methods . In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Lan...

    2018 doi

  4. [12]

    • Input: ‘Financial managers‘ • Output: ‘Financial manager oversees budget- ing and financial planning.‘

    Single Occupation: If the input contains a single occupation, rephrase it to include a brief action related to the job. • Input: ‘Financial managers‘ • Output: ‘Financial manager oversees budget- ing and financial planning.‘

  5. [13]

    • Input: ‘Models, demonstrators, and product promoters‘ • Output: ‘Model showcases clothing and ac- cessories.‘

    Multiple Occupations: If the input contains multiple occupations, choose the most common or relevant one and rephrase it similarly. • Input: ‘Models, demonstrators, and product promoters‘ • Output: ‘Model showcases clothing and ac- cessories.‘

  6. [14]

    Simplifying Categories: For broad or com- pound occupation categories, select the most iden- tifiable and relevant job title and provide a brief action description. • Input: ‘Conveyor, dredge, and hoist and winch operators‘ • Output: ‘Conveyor operator manages con- veyor syste...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.