REVIEW 5 major objections 5 minor 19 references
Fairness Dynamics During Training
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Tracking token-level fairness during Pythia-6.9b pretraining shows the model becomes increasingly confident and accurate at predicting 'male' over 'female' after about 80k steps, and early stopping there trades 1.7% LAMBADA accuracy for a…
desk verdict Descriptive bias trajectories are solid; the early-stopping gain is a selected maximum, not an intervention estimate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a pair of token-level metrics applied at every training checkpoint. Average Rank (AR) records the mean rank of the correct token's probability across the full vocabulary, so a drop in rank means the model is getting closer to the right answer even when accuracy is still near zero. Jensen-Shannon Divergence by Parts (JSD-P) computes, separately for each answer option, the divergence between the model's output probability and the ideal one-hot distribution, so the male/female gap in JSD-P is read as bias and its magnitude as confidence. These two metrics, plotted across Pythia checkpoints, are what let the authors see the post-80k divergence that accuracy and Stereotype Accuracy hide.
What would settle it
Recompute the early-stopping gain on a held-out split of WinoBias items not used to select the 80k checkpoint, or with a decision rule fixed in advance; if the fairness gain shrinks substantially or the 1.7% LAMBADA cost grows, the early-stopping claim fails. A second decisive check would be to rerun the male/female JSD-P comparison on prompts balanced for occupation frequency and difficulty; if the gap disappears, the result reflects task difficulty rather than bias.
Extended reading notes
Core claim
On WinoBias Type 2 gender-prediction prompts, Pythia-6.9b's fairness trace does not mirror its performance trace: the male/female gap in JSD-P stays small until roughly 80k steps, then widens as training continues, while the average rank of the correct token improves faster when the answer is 'male' and confidence in gendered answers grows. The paper's central claim is that this checkpoint-level signature indicates a real bias toward male predictions, and that an early-stopping intervention at that step yields a model whose mean JSD-P difference between male and female drops from 0.73 to 0.05, a 92.5% fairness improvement, while LAMBADA accuracy falls by only 1.7 percentage points.
Load-bearing premise
The load-bearing premise is that the roughly 80k-step checkpoint is a genuine intervention point rather than a posterior selection, because the same JSD-P fairness curve and the same evaluation set are used both to find the step and to measure the 92.5% gain, with no held-out validation or pre-specified stopping rule.
Editorial extensions
If this is right
- Early stopping at a fairness-optimal checkpoint becomes a concrete training intervention: for Pythia-6.9b it trades 1.7% LAMBADA accuracy for a 92.5% reduction in the male/female JSD-P gap.
- Fairness should be monitored as a training trajectory, since the paper shows bias can emerge suddenly at a step where conventional performance metrics give no warning.
- All-or-nothing metrics can mislead: Accuracy and Stereotype Accuracy miss the growing confidence asymmetry that JSD-P exposes between male and female answers.
- Larger models may be more prone to gendered assumptions in neutral contexts, as Pythia-6.9b assigns more probability mass to 'male' and 'female' when 'not specified' is correct than Pythia-160m does.
Reading between the lines
- My reading extends beyond the paper: the 92.5% figure is likely an optimistic upper bound, because the 80k stopping point is selected by inspecting the same JSD-P curve on the same evaluation set that defines the gain, without a held-out validation split or a predetermined decision rule.
- If the pattern transfers, AR and JSD-P could serve as early-warning signals for other bias axes (e.g., race or age) during pretraining, provided the evaluation prompts are matched for difficulty and lexical frequency.
- A testable extension would be to apply the same checkpoint monitoring to Pythia-160m and other model families to see whether the post-80k male/female divergence is a general scaling phenomenon or specific to 6.9b.
- The paper's own limitation note implies that early stopping only sidesteps bias on binary gender; a stronger intervention would need to alter the training signal itself at the divergence point.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper monitors gender-prediction fairness during pretraining of Pythia models using two proposed metrics, Average Rank (AR) and Jensen-Shannon Divergence by Parts (JSD-P), computed from next-token probabilities on WinoBias prompts. It reports that Pythia-6.9b becomes increasingly biased toward male predictions after roughly 80k steps; that early stopping at this point trades 1.7% LAMBADA accuracy for a 92.5% fairness improvement; and that larger Pythia models make more gendered assumptions in gender-neutral contexts. The authors support these claims with plots of AR and JSD-P over training and Mann-Whitney U tests across five random seeds.
Significance. The descriptive finding that a fairness gap can emerge abruptly and diverge from conventional accuracy curves is potentially valuable, and the proposed AR/JSD-P metrics are clearly defined and more informative than accuracy or stereotype accuracy in the examples shown. The manuscript openly uses the Pythia suite and WinoBias, making the empirical setup reproducible in principle. However, the central intervention claim—early stopping at ~80k yields a 92.5% fairness gain—is currently a post hoc selection on the evaluation curve used to define the gain, and the bias interpretation is clouded by unmatched answer classes. With additional validation or a reframing as a descriptive observation rather than a mitigation prescription, the work could make a useful contribution.
major comments (5)
- [Section 1, Fig. 1(b)] The 92.5% fairness improvement and associated 1.7% LAMBADA loss are computed at the checkpoint (~80k) that minimizes the same JSD-P fairness curve on the same evaluation set. Because no held-out checkpoint-selection rule, pre-registered stopping criterion, or correction for examining multiple checkpoints is provided, the reported gain is the selected optimum of this one trajectory rather than an estimate of what early stopping would deliver. Please validate the stopping rule on a held-out split or reframe the result as a post hoc observation.
- [Section 2] The male/female comparison pools WinoBias prompts with different occupations, pronoun targets, and answer-token base rates. Differences in AR or JSD-P between "male" and "female" could therefore reflect unequal task difficulty or lexical frequency rather than gender bias. Matching answer classes or reporting per-occupation results would strengthen the bias interpretation.
- [Appendix D, Figs. 4(b), 5(b), 6(b)] The Mann-Whitney U tests pool repeated evaluations of the same prompts across five seeds; these observations are not independent, so the resulting p-values overstate significance. Please use a cluster-robust test or treat seeds as the unit of analysis.
- [Section 2, Figs. 1(b), 1(c), and Appendix C] The main results are shown only on the pro-stereotype split of WinoBias. Fairness dynamics and the male/female divergence may differ on the anti-stereotype split, which is standard in WinoBias evaluations; reporting both splits is necessary for the claim that fairness declines during training.
- [Appendix B] The use of the token "not" as a proxy for the phrase "not specified" is stated to be justified by high probability continuity, but no quantitative evidence is given. Please report the rate at which "specified" follows "not" for the relevant prompts, or compute the metric using the full phrase.
minor comments (5)
- [Appendix B, Eq. (1)] Equation (1) uses A(i)j and B(i)j without definitions and does not appear to be used in Eq. (2); please remove it or define the notation.
- [Appendix D] The text says "For the figures presented in the main body, Figs. 4(a), 5(a), and 6(a)" but these figures are in the appendix, not the main body; renumber or reword.
- [Section 1] "Open AI's LAMBADA benchmark" misattributes LAMBADA; it should cite Paperno et al. (2016) as the source.
- [Fig. 3] The text says SA scores of 1 and 0 are most biased and 0.5 is least biased; this direction should be stated in the caption or text for clarity.
- [Appendix D] Throughout, "p < 0.01" is reported without exact p-values or effect sizes; reporting these would aid interpretation.
Circularity Check
The 92.5% early-stopping fairness gain is a post hoc selection from the same WinoBias JSD-P curve used to measure it.
-
fitted input called prediction
[Section 2 (Approach), sentence after Figs. 1(b), 1(c); Fig. 1(b) caption]
"By tracking AR and JSD-P during training, in Figs. 1(b), 1(c), we establish that Pythia-6.9b can benefit from early stopping at ≈80 k steps, trading a 1.7% accuracy decrease on LAMBADA [15] for a 92.5% increase in fairness (mean JSD-P difference drops from 0.73 to 0.05). Fig. 1(b) caption: 'Mean JS Divergence by Parts: Correct Answer - Gender Disambiguated, Prostereotype Split, 6.9b 80k'."
The reported fairness gain is computed at the checkpoint that minimizes the very JSD-P gap used to define the gain: the paper marks '80k' on the same male/female JSD-P curve and then reports the drop from 0.73 to 0.05 at that point. Because the stopping step is selected by inspecting the evaluation curve on the same WinoBias set, the 92.5% figure is the selected optimum, not an estimate of what an early-stopping rule would deliver. No held-out checkpoint-selection rule or pre-registered decision rule is provided, and the LAMBADA cost is also measured at that selected step.
full rationale
The paper's descriptive results are self-contained and not circular: AR and JSD-P are computed from external Pythia checkpoints and WinoBias prompts, and the observations that male/female JSD-P diverges after ~80k steps and that 6.9b assumes gender more than 160m do not depend on the paper's own fitted values. The central circularity is confined to the early-stopping claim. The 92.5% fairness gain is measured at the checkpoint where the same JSD-P gap is smallest on the same evaluation data, so the reported benefit is selected rather than predicted; no held-out checkpoint selection is described. Appendix A's limitation that only binary gender is evaluated and that early stopping 'simply works around bias' is acknowledged, but it does not address this selection problem. Because a substantial independent descriptive analysis remains, the paper is partially circular rather than fully reducible to its inputs, supporting a score of 6.
Assumptions & free parameters
free parameters (1)
- Early stopping step (~80k) =
~80k training steps
assumptions (4)
- domain assumption Pythia model checkpoints at each training step are representative samples of LLM training behavior
- domain assumption The three-option prompt (male, female, not specified) on WinoBias Type 2 measures gender bias in occupation prediction
- ad hoc to paper The answer token 'not' is an adequate proxy for the phrase 'not specified'
- domain assumption JSD-P per answer option is a valid fairness metric
Cite this review
Pith. "Pith review of Fairness Dynamics During Training." pith.science (2026). https://pith.science/paper/7RKPCNCX
@misc{pith2026250601709,
author = {Pith},
title = {Pith review of: Fairness Dynamics During Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/7RKPCNCX}},
note = {Machine review of arXiv:2506.01709}
}
read the original abstract
We investigate fairness dynamics during Large Language Model (LLM) training to enable the diagnoses of biases and mitigations through training interventions like early stopping; we find that biases can emerge suddenly and do not always follow common performance metrics. We introduce two new metrics to evaluate fairness dynamics holistically during model pre-training: Average Rank and Jensen-Shannon Divergence by Parts. These metrics provide insights into the Pythia models' progression of biases in gender prediction of occupations on the WinoBias dataset. By monitoring these dynamics, we find that (1) Pythia-6.9b is biased towards men; it becomes more performant and confident predicting "male" than "female" during training, (2) via early-stopping, Pythia-6.9b can exchange 1.7% accuracy on LAMBADA for a 92.5% increase in fairness, and (3) larger models can exhibit more bias; Pythia-6.9b makes more assumptions about gender than Pythia-160m, even when a subject's gender is not specified.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Pythia: A suite for analyzing large language models across training and scaling
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. InInternational Conference on Machine Learning, pages 2397–2430. PMLR, 2023
2023
-
[2]
Pretrained Language Model Embryology: The Birth of ALBERT
Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. Pretrained language model embryology: The birth of albert.arXiv preprint arXiv:2010.02480, 2020
work page Pith review arXiv 2010
-
[3]
Theories of “gender” in nlp bias research
Hannah Devinney, Jenny Björklund, and Henrik Björklund. Theories of “gender” in nlp bias research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 2083–2102, New York, NY , USA, 2022. Association for Computing Machinery
work page 2022
-
[4]
On the impact of machine learning randomness on group fairness
Prakhar Ganesh, Hongyan Chang, Martin Strobel, and Reza Shokri. On the impact of machine learning randomness on group fairness. InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1789–1800, 2023
work page 2023
-
[5]
Bigbench: Towards an industry standard benchmark for big data analytics
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. InProceedings of the 2013 ACM SIGMOD international conference on Management of data, pages 1197–1208, 2013
work page 2013
-
[6]
Towards understanding fairness and its composition in ensemble machine learning
Usman Gohar, Sumon Biswas, and Hridesh Rajan. Towards understanding fairness and its composition in ensemble machine learning. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1533–1545. IEEE, 2023
work page 2023
-
[7]
Ffb: A fair fairness benchmark for in-processing group fairness methods, 2024
Xiaotian Han, Jianfeng Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, and Xia Hu. Ffb: A fair fairness benchmark for in-processing group fairness methods, 2024
work page 2024
-
[8]
Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. Debiasing isn’t enough!–on the effectiveness of debiasing mlms and their social biases in downstream tasks.arXiv preprint arXiv:2210.02938, 2022
work page Pith review arXiv 2022
Show all 19 references
-
[9]
Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
2001 arXiv
-
[10]
J. Lin. Divergence measures based on the shannon entropy.IEEE Transactions on Information Theory, 37(1):145–151, 1991
1991
-
[11]
Probing across time: What does roberta know and when?arXiv preprint arXiv:2104.07885, 2021
Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when?arXiv preprint arXiv:2104.07885, 2021
2021 arXiv
-
[12]
On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statistics, pages 50–60, 1947
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statistics, pages 50–60, 1947
1947
-
[13]
Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020
2004 arXiv
-
[14]
Crows-pairs: A challenge dataset for measuring social biases in masked language models.arXiv preprint arXiv:2010.00133, 2020
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models.arXiv preprint arXiv:2010.00133, 2020
2010 arXiv
-
[15]
The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016
2016 arXiv
-
[16]
Are emergent abilities of large language models a mirage?, 2023
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?, 2023
2023
-
[17]
Fewer errors, but more stereotypes? the effect of model size on gender bias.arXiv preprint arXiv:2206.09860, 2022
Yarden Tal, Inbal Magar, and Roy Schwartz. Fewer errors, but more stereotypes? the effect of model size on gender bias.arXiv preprint arXiv:2206.09860, 2022
2022 arXiv
-
[18]
Training trajectories of language models across scales.arXiv preprint arXiv:2212.09803, 2022
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales.arXiv preprint arXiv:2212.09803, 2022
2022 arXiv
-
[19]
not specified
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods.CoRR, abs/1804.06876, 2018. 3 A Limitations and Social Considerations Our evaluation is limited to the WinoBias dataset and the Py...
2018 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.