REVIEW 6 major objections 5 minor 3 references
Gender Inequality in English Textbooks Around the World: an NLP Approach
T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read English textbooks in all seven cultural spheres systematically overrepresent male characters.
desk verdict A genuinely cross-cultural dataset and question, undermined by a keyword list that cannot separate generic from gendered use and a statistical model that treats double-counted segments as independent draws. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is a hand-built gendered keyword list, Table 2, pairing male and female terms such as “he/she,” “father/mother,” and “king/queen.” Text is cut into 100-character windows; any window containing a male keyword goes into the male document, and any window containing a female keyword goes into the female document, so a window can enter both. Count is the number of segments per document tested against a Bernoulli 0.5 null; firstness is counted only when matched male/female pairs occur adjacent or with one stopword between them. TF-IDF then produces the two gendered word lists, names in those lists are counted by hand, LLMs are asked to tell the lists apart, and GloVe cosine distances link the lists to stereotype-relevant keywords.
What would settle it
Annotate a random sample of the male-keyword segments and classify every occurrence of “he,” “his,” “man,” and “men” as either specifically male or generic; if a substantial share are generic uses, the male advantage in count, firstness, and TF-IDF would be partly an artifact of the keyword list.
Extended reading notes
Core claim
The paper's central claim is that gender inequality is pervasive in English textbooks around the world, in all cultural spheres, with more male count, firstness, and names. On the paper's own terms, the discovery is a cross-cultural regularity: measured over 100-character segments that contain gendered keywords, male-keyword segments outnumber female-keyword segments overall (4,764 vs 3,740, $p \approx 10^{-28}$), male firstness outnumbers female firstness overall (247 vs 183, $p < 0.01$), and male names outnumber female names in the TF-IDF lists (62 vs 23). No cultural sphere reaches parity on all indicators; the Latin American sphere is closest on count and firstness.
Load-bearing premise
The hand-built keyword list assumes that words like “he,” “his,” and “man” mark a male character whenever they appear, rather than serving as generic or epicene references in many sentences.
Editorial extensions
If this is right
- Because count, firstness, and name data all skew male, a student reading these textbooks is systematically more likely to encounter male characters, see males named first in paired mentions, and meet male proper names.
- The Latin American cultural sphere is the closest to equal counts and firstness; if the paper is right, its textbooks are the least male-skewed among the seven, while the Western European sphere shows the strongest count imbalance.
- Male firstness is statistically significant overall and in the Eastern European and Islamic spheres, even though most individual spheres lack enough paired mentions to reach significance.
- TF-IDF word lists are distinct enough that LLMs can usually tell which list is male and which is female, meaning the vocabulary surrounding each gender is stereotypically separated.
- GloVe distances show that words like “pretty,” “baby,” “love,” and “nurse” sit closer to the female word cluster and “death” closer to the male cluster, in line with broader corpus gender associations.
Reading between the lines
- A direct test of the keyword assumption would be to annotate a random sample of the male-keyword segments and ask whether each “he,” “his,” “man,” and “men” refers to a specific male person; if many are generic or epicene uses, the male counts are inflated even if the direction likely persists.
- The same pipeline could be run at country level rather than sphere level, which might reveal that publisher origin, rather than cultural sphere, drives the imbalance; several Latin American textbooks come from Western publishers.
- The method could be translated to textbooks in other languages and applied to other subjects such as history or mathematics, asking whether English class is more or less gendered than the rest of the curriculum.
- A future replication could test whether the LLM recognition result survives when the lists are scrambled or when the prompt is varied, since the paper's trials all follow the same prompt structure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper uses NLP methods to quantify gender inequality in English textbooks from 22 or 23 countries grouped into seven cultural spheres. It constructs male and female keyword lists, splits cleaned text into 100-character segments, and assigns each segment to the male and/or female document depending on keyword presence. From these documents it computes segment counts, firstness ratios, TF-IDF word lists, manual name counts, LLM recognition of the two word lists, and GloVe distance of selected keywords to the two lists. The central claims are that male characters are overrepresented in count, firstness, and names across all spheres, and that the Latin cultural sphere shows the least inequality.
Significance. If the results were valid, a cross-cultural, corpus-scale comparison of gender inequality in textbooks would be a useful contribution to the education and NLP literature. The paper assembles a rare multi-country corpus and attempts several complementary metrics, which is commendable. However, the validity of every headline result depends on two assumptions that the manuscript does not establish: that the hand-built keyword list isolates gender-specific references rather than generic uses, and that the segment-counting procedure supports the stated Bernoulli model. The internal contradictions in the reported p-values and the preprocessing ambiguity further undermine confidence. With substantial re-analysis the cross-cultural dataset could still support a defensible descriptive study, but the current evidence does not reliably support the stated conclusions. I also note that the manuscript does not provide code or data availability statements, which limits reproducibility.
major comments (6)
- [Methods (Count), Table 4] The p-values in Table 4 are computed under a Bernoulli(0.5) model, but the counting unit is the 100-character segment, and a segment containing keywords of both genders is saved into both documents (Methods, Data Cleaning). Male and female counts are therefore not mutually exclusive, and within a textbook the segments are not independent Bernoulli draws. As a consequence, the z-test p-values in Table 4 do not have their stated interpretation; for example, a segment like "boys and girls" increments both counts. The analysis should use a unit of analysis that respects this structure, such as classifying each segment as male-only, female-only, both, or neither and using a cluster-robust method at the textbook level.
- [Methods (Table 2)] The keyword list in Table 2 contains "he", "his", "him", "man", and "men", all of which have standard generic or epicene uses in English. Because every occurrence of such a token places the entire segment in the male document, even a modest rate of generic usage will inflate every male-side result in Tables 4-8 and the TF-IDF lists. The manuscript never validates the keyword set against generic usage, and the limitation section does not mention it. Please provide evidence, for example manual annotation of a sample of "he" and "man" hits, or restrict the keyword set to unambiguous gender-specific terms before re-running all downstream analyses.
- [Data Cleaning, Table 2] There is a direct contradiction between Data Cleaning and Table 2. The text states that texts are cleaned with NLTK to "remove stopwords," and NLTK's standard English stopword list includes he, him, his, she, and her, yet Table 2 counts these pronouns among the gender keywords. If the stopwords are removed before segmentation, the pronoun keywords cannot fire at all; if they are not removed, the preprocessing description is wrong. Please state the exact pipeline and report token-frequency statistics for each keyword in Table 2.
- [Results (Count), Table 4] The text immediately below Table 4 says that the Sinosphere and Latin Cultural Sphere are the only exceptions, with p-values of 0.086 and 0.368, but Table 4 itself reports 0.0341 and 0.2702. Under the table's two-tailed test the Sinosphere result is significant at the 0.05 level, so the statement that "only two exceptions" is inconsistent with the reported data. This discrepancy must be resolved, and the conclusion about which spheres show significant imbalance must be recalculated.
- [Discussion, Conclusion] The conclusion that the Latin cultural sphere shows the least inequality is not established by the reported tests. The supporting evidence is a nonsignificant within-sphere p-value for count (Table 4: p=0.2702) and firstness (Table 5: p=0.66), but absence of evidence for imbalance within a small sample is not evidence of less imbalance than other spheres. The paper needs a formal cross-sphere comparison, such as a logistic regression with sphere as a predictor or an explicit test for interaction, and it should correct for multiple comparisons across the seven spheres.
- [Results (LLM Recognition, Distance to Keywords), Tables 6-8] The LLM recognition and GloVe distance results do not provide independent evidence about the textbooks. Both the LLMs and GloVe are trained on large general-language corpora that already encode societal gender associations, so their ability to label the TF-IDF lists or place words closer to one gender may reflect the training data rather than anything specific to these textbooks. If these analyses are retained, they should be reframed as comparisons between textbook word lists and general-corpus associations, and the small GloVe differences in Table 8 (0.01-0.05) should be accompanied by uncertainty estimates or significance tests.
minor comments (5)
- [Abstract, Methods, Table 1] The abstract says 22 countries, while the Methods section says 23 countries and regions; Table 1 lists "Southern Cone" as a row, which is a region rather than a country, and "Philippians" should be "Philippines". Please reconcile these counts and labels.
- [Methods (Firstness), Conclusion] The firstness definition requires only one stopword between gendered terms, but because stopwords are removed earlier, the example "ladies and gentleman" would become "ladies gentleman" after cleaning; please clarify how the "one stopword" criterion is applied to the cleaned text.
- [Tables 4 and 5] Tables 4 and 5 report p-values to varying precision and do not state the total number of tests or any multiple-comparison correction; given seven spheres plus the overall test, please report adjusted p-values or explicitly state that no correction was applied.
- [Discussion (Counting Names)] The Monte Carlo check on names is described only as "It turns out that there are no names," which is too vague to evaluate; please provide the procedure, the null distribution, and the numeric result.
- [Figures] The manuscript refers to Images 1-4, but these images are not embedded in the text; they are necessary for interpreting the bar charts and should be included or clearly referenced as supplementary files.
Circularity Check
No significant circularity: the central counts and firstness results are direct parameter-free tallies from textbook segments, not outputs of a fitted model or of the paper's own prior claims.
full rationale
The central results (Table 4 counts and Table 5 firstness) are obtained by direct segmentation and counting of the textbooks using a fixed, disclosed keyword list (Table 2); no parameter is fitted to the target labels and no quantity is predicted from a fitted version of itself. The TF-IDF, name-count, LLM-recognition, and GloVe-distance sections are secondary analyses whose inputs (the two gendered documents) are derived from the same keyword-segmentation scheme, but they do not feed back into the headline counts. The LLM and GloVe checks use external, pre-trained models not trained on this paper's data, so they are independent evidence rather than circular validation. No load-bearing self-citation chain exists: the only citations are to external prior work (e.g., Huntington 1993; Pennington et al. 2014). A genuine reproducibility issue is that Data Cleaning removes NLTK stopwords while Table 2 lists pronoun keywords such as 'he', 'him', 'his', 'she', 'her'; if those pronouns are removed before segmentation, the effective keyword set is not the one printed and the exact counts in Tables 4-5 are under-specified. That is a measurement-validity and reproducibility concern, not circularity, because the reported numbers are still direct counts rather than quantities that reduce to their own inputs. Likewise, the possible generic/epicene use of 'he' and 'man' would question construct validity, but it would not make the derivation circular under the definitions the paper states. Therefore no circular step is identifiable from the text.
Assumptions & free parameters
free parameters (5)
- context_segment_length =
100 characters
- firstness_gap_limit =
1 stopword
- tfidf_top_n =
300
- glove_difference_threshold =
0.03
- significance_threshold =
0.01
assumptions (5)
- domain assumption The gendered keyword list (Table 2) unambiguously identifies male and female references, including 'he', 'his', 'man', and 'men' as male-specific.
- domain assumption Segments are independent Bernoulli trials with P=0.5 for male versus female under the null.
- domain assumption Huntington's Clash of Civilizations cultural spheres are a valid grouping for textbook gender inequality.
- domain assumption LLM ability to distinguish TF-IDF lists reflects real differences in textbook language, not the model's own prior gender bias.
- domain assumption GloVe embeddings are an appropriate external benchmark for real-world gender bias.
Cite this review
Pith. "Pith review of Gender Inequality in English Textbooks Around the World: an NLP Approach." pith.science (2026). https://pith.science/paper/6VFQYQ26
@misc{pith2026250602425,
author = {Pith},
title = {Pith review of: Gender Inequality in English Textbooks Around the World: an NLP Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/6VFQYQ26}},
note = {Machine review of arXiv:2506.02425}
}
read the original abstract
Textbooks play a critical role in shaping children's understanding of the world. While previous studies have identified gender inequality in individual countries' textbooks, few have examined the issue cross-culturally. This study applies natural language processing methods to quantify gender inequality in English textbooks from 22 countries across 7 cultural spheres. Metrics include character count, firstness (which gender is mentioned first), and TF-IDF word associations by gender. The analysis also identifies gender patterns in proper names appearing in TF-IDF word lists, tests whether large language models can distinguish between gendered word lists, and uses GloVe embeddings to examine how closely keywords associate with each gender. Results show consistent overrepresentation of male characters in terms of count, firstness, and named entities. All regions exhibit gender inequality, with the Latin cultural sphere showing the least disparity.
Reference graph
Works this paper leans on
-
[1]
Gender inequality exists in English textbooks in general. 2. Gender inequality is presented differently in the English textbooks, depending on the cultural sphere that the source country of the textbooks. The way that this study divides up cultural spheres is adopted is from Huntington’s The Clash of Civilizations . The cultural spheres include:
-
[2]
Indosphere: influenced by Indian Culture 3
Sinosphere: influenced by Chinese culture 2. Indosphere: influenced by Indian Culture 3. Islamic cultural sphere: muslim countries that are in middle east and North Africa 4. African cultural sphere: Mainly in the South of Africa 5. Western cultural sphere: West Europe, North America, and some Oceania countries
-
[6]
Eastern Europe cultural sphere: East Europe 7. Latin American cultural sphere: Mexico and Most countries in South America.(Huntington, 1993) There have been previous studies that demonstrated that LLMs have hidden gender bias due to its capturing of the gender biases in the corpus that it is trained on. Although LLM companies have made the effort to regul...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.