REVIEW 4 major objections 5 minor 2 cited by
Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Prompt-adapted LLMs inherit the gender bias of their pre-trained base models.
desk verdict Strong prompt-adaptation bias transfer result, but the headline claim about pre-trained models is not supported by the experiments, which use instruction-tuned checkpoints. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Selection Bias (SB) metric, designed so that intrinsic and prompted bias can be compared on the same scale. For intrinsic evaluation, the model's sum of token log-probabilities for the referent and distractor occupations decides the answer; for prompted evaluation, the next generated token picks among the referent, distractor, and 'Unknown' options in a randomized multiple-choice prompt. The transfer test is the Pearson correlation between per-occupation SB in the intrinsic setting and per-occupation SB in the prompted setting. WinoBias supplies 3,160 balanced sentences split into syntactically ambiguous (Type 1) and unambiguous (Type 2) items, and each prompting experiment is run over five random seeds with shuffled option order to reduce format noise.
What would settle it
Re-run the correlation with the distractor's position in the multiple-choice options randomized blockwise; if placing the distractor first reliably lowers the Pearson coefficient from $\rho \ge 0.94$ to near zero on the same occupations, the transfer claim would be an answer-order artifact. A second check: measure intrinsic and prompted SB on a set of occupations not in WinoBias; if a model with high intrinsic SB shows near-zero prompted SB there, transfer would not generalize beyond the benchmark.
Extended reading notes
Core claim
The central discovery is that the bias transfer hypothesis holds for prompt-adapted causal LLMs when intrinsic and prompted biases are measured with the same quantity, the Selection Bias (SB) metric. SB is the difference between the rate at which an occupation is generated when the pronoun is male and the rate when it is female; the paper computes this per occupation and correlates the intrinsic and prompted versions across occupations. Under this metric, intrinsic bias in the pre-trained model strongly predicts zero-shot and few-shot prompted bias, with Pearson coefficients $\rho \ge 0.94$ and negligible $p$-values. Explicit pre-prompts that instruct fairness or bias shift overall bias levels but leave the correlation essentially intact ($\rho \ge 0.92$), and few-shot context length and stereotypical makeup also do not break it ($\rho \ge 0.97$). The paper concludes that biases do transfer under prompting, making pre-trained model fairness a practical determinant of downstream prompt-adapted model fairness.
Load-bearing premise
The result assumes that Selection Bias measured by token log-probability sums in the intrinsic setting and by next-token choice in the prompted setting capture the same underlying bias, rather than two measurements that share an artifact such as occupation-name tokenization or answer-order effects.
Editorial extensions
If this is right
- Zero- and few-shot prompting do not dilute the gender bias already present in the pre-trained model; the base model's per-occupation bias predicts the prompted model's bias with $\rho \ge 0.94$.
- Fairness work on the pre-trained model is therefore a practical lever for fairness in prompt-deployed systems: prompt-level instructions shift the level of bias but leave the correlation with intrinsic bias intact.
- Few-shot prompt design—how many examples are used and whether they are pro-stereotypical, anti-stereotypical, or neutral—does not change the strength of bias transfer ($\rho \ge 0.97$).
- The finding contrasts with earlier bias-transfer studies on fine-tuned masked language models, implying that the adaptation strategy (fine-tuning versus prompting) is a decisive variable in whether intrinsic bias transfers.
Reading between the lines
- If the correlation is as stable as reported, an intrinsic audit (token log-probability scoring) could serve as a cheap screening step for likely deployed fairness before running full behavioral evaluations of prompt-adapted models.
- The same experimental design could be pushed to other causal adapters—low-rank adapters, full fine-tuning, chain-of-thought prompting—to map where transfer finally breaks; earlier fine-tuning results suggest it might.
- Because Section 3.2 defines SB as an absolute difference while the figures plot signed values, re-running the analysis with signed per-occupation bias would show whether the direction, not just the magnitude, of bias transfers; a flip in direction would change the fairness prescription.
- Extending beyond binary gender to gender-neutral pronouns is the paper's own stated limitation; a dataset that unambiguously marks singular 'they' would test whether the strong transfer persists outside the binary.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether gender bias transfers from pre-trained causal language models to prompt-adapted versions. Using WinoBias pronoun co-reference resolution, the authors define an intrinsic next-token-generation evaluation and zero- and few-shot multiple-choice prompting evaluations, and measure a per-occupation Selection Bias (SB) metric. They report strong Pearson correlations (rho >= 0.94) between intrinsic and prompted biases for Llama 3 8B/70B, Falcon 40B, and Mistral 7B, and correlations remain high (rho >= 0.92) when models are pre-prompted with fairness- or bias-inducing prompts and when few-shot composition is varied. The paper concludes that pre-trained model fairness is a strong predictor of prompt-adapted bias, in contrast to prior findings for fine-tuned masked language models.
Significance. If the central claim were established for base pre-trained checkpoints, the result would be an important contribution to the bias transfer literature: it would show that, unlike fine-tuning adaptation, prompt adaptation preserves intrinsic bias, making pre-training fairness critical for downstream deployments. The paper has real strengths: it proposes a unified metric for intrinsic and prompted evaluations, evaluates several model families at multiple scales, averages over random seeds, randomizes answer ordering, and is transparent about the limitation to WinoBias and binary gender. The high correlations on the instruction-tuned checkpoints actually used are themselves interesting. However, the abstract and conclusion make a stronger claim about pre-trained models than the experimental setup in Section 3.1 supports, and the ambiguity in the SB definition affects the interpretation of every correlation coefficient. These issues are fixable but currently prevent the headline result from being accepted as stated.
major comments (4)
- [Section 3.1, Tables 1-2, Abstract and Section 6] The experiments use instruction fine-tuned versions of the models, but the abstract claims to show that 'intrinsic biases in pre-trained Mistral, Falcon and Llama models' correlate with prompted biases. Section 3.1 states: 'we investigate fairness in adaptations using the instruction fine-tuned versions of highly performant LLMs.' The intrinsic evaluations in Table 1 are next-token-generation measurements on these instruction-tuned checkpoints, so they reflect an already-adapted model, not the pre-trained checkpoint. Consequently, the reported rho values do not establish transfer from pre-training to prompt adaptation. The authors must either rerun the intrinsic and prompted evaluations on the base (non-instruction-tuned) checkpoints, or substantially revise the abstract, introduction, and conclusion to claim only that biases in instruction-tuned models predict their prompted behavior. The captions should also identify the exact checkpoints used (for example, Mistral-Instruct version, Falcon-Instruct version, and Llama 3 base vs. Instruct variants).
- [Section 3.2, Figures 2 and 5-9] Selection Bias is defined in Section 3.2 as 'the absolute difference in rates that an occupation is generated by a model when a male pronoun is present in a sentence vs. a female pronoun.' However, Figures 5-9 plot signed values, with the captions explaining that 'less than zero is female-biased and greater than zero is male-biased,' and Figure 2 describes biases as 'directionally aligned.' Pearson correlations between per-occupation SB values depend critically on whether the signed or absolute formulation is used; absolute values would lose direction and could inflate positive correlation through shared magnitude effects. The paper should state explicitly which definition was used for the correlations in Table 2 and Figure 3, define the signed SB formally (for example, as P(occupation | male pronoun) - P(occupation | female pronoun)), and, if advisable, report both signed and absolute correlations.
- [Section 4.2 and Appendix E] The claims about bias transfer under fairness- and bias-inducing pre-prompts are based only on the most effective positive pre-prompt and the most effective negative pre-prompt, selected 'in an ad-hoc and iterative way' from a larger set (Appendix E). Because the selection criterion is the same SB that is later correlated, the correlations in Table 2 are conditional on this outcome-based selection and are not corrected for multiple comparisons. This makes the 'bounds of bias transfer' claim in Section 4.2 exploratory rather than confirmatory. The authors should report results for all prompts in the appendix, or pre-specify the prompt set, and show the distribution of correlations across prompts.
- [Section 4.3 and Table 4] The claim that 'increasing number of pro-stereotypical ambiguous samples incrementally worsens SB' is not supported by the data in Table 4: the pro-stereotypical mean SB values are 23.86, 22.77, 25.29, and 23.71 for 3, 10, 20, and 30 shots, respectively, which is not a monotonic increase. If the intended claim concerns only ambiguous (Type 1) sentences, the table does not provide the Type 1/Type 2 split for this experiment. Please clarify the claim or provide the supporting per-type numbers.
minor comments (5)
- [Section 4.2] The paper states that for Llama 3 8B with negative pre-prompts, 'its guardrails are triggered for nearly 4% of the dataset,' but does not describe how these cases are handled in the RPA and SB computations. Please state whether refusals are counted as incorrect answers, excluded from the denominator, or mapped to 'Unknown.'
- [Tables 1, 2, 4 and Figure 2] Pearson correlations are reported without confidence intervals, and Table 4 reports only 'p approximately zero' rather than exact p-values. Given that the correlations are computed over a fixed set of occupations, confidence intervals would help assess the stability of the rho >= 0.94 claim.
- [Section 4.3 and Abstract] The composition experiments are conducted only on Llama 3 8B, but the abstract states that bias transfer remains strong when few-shot length and stereotypical composition are varied without noting this model restriction. Please scope the claim explicitly or add the analogous experiments for at least one more model.
- [Section 3.1 and Appendix A-D] The model names are inconsistent with the cited references: Table 1 and the appendix refer to 'Mistral 3 7B' and 'Llama 3 8B/70B,' but the cited Mistral paper is 'Mistral 7B' and the cited Llama paper is the original Llama. Please use exact official checkpoint names and versions throughout.
- [Figures 5-9] The legend text for the intrinsic condition says there is no standard deviation 'as there is no stochasticity involved in the next token selection,' but the intrinsic condition is still compared with seeded stochastic prompting conditions; this is helpful but should be stated once in the main text to avoid confusion about whether intrinsic results are aggregated over the same seeds.
Circularity Check
No circularity: the bias-transfer finding is an empirical correlation between two prompt formats applied to the same models, and no fitted parameter or self-citation chain reduces the claimed result to its inputs.
full rationale
The paper's central claim is an empirical correlation between intrinsic and prompted selection biases computed on the same occupation-level WinoBias sentences. Section 3.2 defines Selection Bias uniformly for both intrinsic next-token scoring and MCQ prompting, and Section 4.1 reports Pearson correlations between these independently measured quantities. No parameter is fitted to the reported correlation; no 'prediction' is generated from a fit; and no load-bearing self-citation or imported uniqueness theorem is used. The new SB metric unifies the measurement scale for intrinsic and extrinsic evaluation, but the high correlation is a contingent empirical result about model behavior, not an identity. The only notable weakness is an internal-validity gap: Section 3.1 says instruction-tuned checkpoints were used while the abstract and conclusion say 'pre-trained' biases transfer; that inconsistency affects what the experiments demonstrate, but it is not circular reasoning. Accordingly, no circular steps are identified.
Assumptions & free parameters
free parameters (1)
- Selected fairness-inducing and bias-inducing prompts (best/worst) =
N/A: prompts chosen post hoc from a set of candidates
assumptions (4)
- domain assumption WinoBias labels (pro-stereotypical, anti-stereotypical, ambiguous) reflect the ground truth for gender bias in occupations.
- domain assumption The We're All Equal worldview is assumed: any skew in model behavior across demographic groups is structural bias needing mitigation.
- domain assumption Pearson correlation between occupation-level selection bias vectors is an appropriate measure of bias transfer.
- domain assumption Next-token generation in the prompted MCQ setup is a consistent estimator of the model's preference on WinoBias.
Cite this review
Pith. "Pith review of Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models." pith.science (2026). https://pith.science/paper/COOA6RAR
@misc{pith2026241203537,
author = {Pith},
title = {Pith review of: Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/COOA6RAR}},
note = {Machine review of arXiv:2412.03537}
}
read the original abstract
Large language models (LLMs) are increasingly being adapted to achieve task-specificity for deployment in real-world decision systems. Several previous works have investigated the bias transfer hypothesis (BTH) by studying the effect of the fine-tuning adaptation strategy on model fairness to find that fairness in pre-trained masked language models have limited effect on the fairness of models when adapted using fine-tuning. In this work, we expand the study of BTH to causal models under prompt adaptations, as prompting is an accessible, and compute-efficient way to deploy models in real-world systems. In contrast to previous works, we establish that intrinsic biases in pre-trained Mistral, Falcon and Llama models are strongly correlated (rho >= 0.94) with biases when the same models are zero- and few-shot prompted, using a pronoun co-reference resolution task. Further, we find that bias transfer remains strongly correlated even when LLMs are specifically prompted to exhibit fair or biased behavior (rho >= 0.92), and few-shot length and stereotypical composition are varied (rho >= 0.97). Our findings highlight the importance of ensuring fairness in pre-trained LLMs, especially when they are later used to perform downstream tasks via prompt adaptation.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 2 Pith papers
-
Guiding LLM Decision-Making with Fairness Reward Models
A single process-level reward model, trained on weakly labeled biased versus unbiased reasoning, transfers across tasks and models to reduce equalized odds gaps in LLM decision-making.
-
Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
UCerF scores LLM fairness by both correctness and confidence, and SynthBias provides 31,756 gender-occupation coreference samples for benchmark testing.
Reference graph
Works this paper leans on
-
[1]
E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867,
-
[6]
J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,
-
[8]
S. Goldfarb-Tarrant, R. Marchant, R. M. Sánchez, M. Pandya, and A. Lopez. Intrinsic bias metrics do not correlate with application bias. arXiv preprint arXiv:2012.15859,
arXiv 2012
-
[9]
URL https://arxiv.org/abs/2310.06825. X. Jin, F. Barbieri, B. Kennedy, A. M. Davani, L. Neves, and X. Ren. On transferability of bias mitigation effects in language model fine-tuning. arXiv preprint arXiv:2010.12864,
arXiv 2010
-
[10]
URL https://arxiv.org/abs/2210.02938. M. Kaneko, D. Bollegala, N. Okazaki, and T. Baldwin. Evaluating gender bias in large language models via chain-of-thought prompting. arXiv preprint arXiv:2401.15585,
- [11]
-
[12]
doi: 10.18653/v1/2022.acl-long.247
Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.247. URL https://aclanthology.org/2022.acl-long.247. 8 H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,
arXiv 2022
-
[13]
E. Vanmassenhove, C. Emmery, and D. Shterionov. Neutral rewriter: A rule-based and neural approach to automatic rewriting into gender-neutral alternatives. arXiv preprint arXiv:2109.06105,
Show all 14 references
-
[15]
9 A Selection biases split by task ambiguity Similar to zero-shot biases in Llama 3 8B in Fig
URL http://arxiv.org/abs/1804.06876. 9 A Selection biases split by task ambiguity Similar to zero-shot biases in Llama 3 8B in Fig. 2(a), the model largely exhibits more bias for ambiguous sentences, and biases that are largely directionally aligned for ambiguous and non- ambi...
-
[2017]
Y . T. Cao, Y . Pruksachatkun, K.-W. Chang, R. Gupta, V . Kumar, J. Dhamala, and A. Galstyan. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations. arXiv preprint arXiv:2203.13928,
-
[2018]
S. Feng, C. Y . Park, Y . Liu, and Y . Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv preprint arXiv:2305.08283,
-
[2021]
Delobelle, E
P. Delobelle, E. K. Tokpo, T. Calders, and B. Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisti...
2022
-
[2022]
H. Dawkins. Second order winobias (sowinobias) test set for latent gender bias detection in coreference resolution. arXiv preprint arXiv:2109.14047,
-
[2023]
Brown, B
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,
1901
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.