Bias Unveiled: Investigating Social Bias in LLM-Generated Code

Fazle Rabbi; Jinqiu Yang; Lin Ling; Song Wang

REVIEW 3 major objections 6 minor 1 cited by

Bias Unveiled: Investigating Social Bias in LLM-Generated Code

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that social bias in LLM-generated code is widespread, quantifiable, and largely mitigable through iterative prompting guided by automated fairness tests.

desk verdict Valuable benchmark and framework, but the headline severity and 90% mitigation claims rest on an unverified sensitive/related attribute mapping that an audit must clear. read the letter →

arxiv 2411.10351 v4 pith:IK3QVWNJ submitted 2024-11-15 cs.SE

Lin Ling , Fazle Rabbi , Song Wang , Jinqiu Yang This is my paper · ORCID

classification cs.SE

keywords socialbiascodegenerationfairnesstestingmetamorphiclargelanguagemodelsmitigationpromptengineeringLLM-generated

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that LLMs that write code carry measurable social bias, and that a black-box testing loop can both quantify and reduce it. It introduces Solar, which turns human-centered task descriptions into code prompts and executable test cases, then checks whether generated code changes its output when only a protected attribute such as gender, race, or age changes. Applied to four code-generation models on 343 tasks, the framework reports severe bias in all of them, with GPT-3.5-turbo-0125 scoring a 60.58% Code Bias Score. Feeding the failing test results back as prompts over three iterations drops that score to 8.77% while functional correctness, measured by attribute usage, rises from 66.60% to 85.66%. The point of caring: social bias in generated code is not just a text-generation artifact; it is testable and partly fixable through prompting alone.

What carries the argument

The central object is Solar's test-case generator, built on a domain-specific language (textX), together with the bias definition in Equation 1: a code snippet is biased if changing one protected attribute while holding everything else fixed changes the output. From each task definition, Solar generates a class skeleton and executable tests that instantiate individuals identical except for one sensitive attribute and assert equal outcomes. The Code Bias Score (CBS) is the fraction of biased executable snippets, the Bias Leaning Score with its range (BLS@Range) captures which demographic value a model favors, and Pass@attribute tracks whether the right related attributes are used and sensitive attributes are avoided. This machinery carries the argument because it converts an abstract fairness concern into runnable assertions and a feedback signal for prompt refinement.

What would settle it

Run Solar on a task where the protected attribute is the actual eligibility criterion, for example a function that grants senior benefits only to people over 60. A deliberately correct implementation would still fail Equation 1 for the age attribute, so the Code Bias Score would not be zero; observing that would show the metric cannot distinguish legitimate attribute use from social bias.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that social bias in LLM-generated code is widespread and quantifiable, and that iterative prompting guided by automated fairness testing can mitigate it without sacrificing functional correctness. The paper supports this with a dataset of 343 real-world human-centered coding tasks in seven categories, a framework that generates prompts and test cases from task definitions, and the Code Bias Score, Bias Leaning Score, and Pass@attribute metrics. The headline numbers are that all four subject models have overall Code Bias Scores above 28%, with GPT-3.5-turbo-0125 highest at 60.58%, and that after three Solar-guided iterations the same model's Code Bias Score falls to 8.77% while Pass@attribute rises from 66.60% to 85.66%. This establishes that a black-box testing pipeline can serve as both a bias detector and a mitigation engine for code-generation models.

Load-bearing premise

The entire measurement rests on treating any difference in output caused by changing a protected attribute as bias; if a task legitimately requires that attribute, such as age for senior health care or employment status for unemployment benefits, the metric will count correct code as biased and the 'severe bias' finding is inflated.

Editorial extensions

If this is right

Social bias in LLM-generated code is measurable by black-box testing, so models can be screened for bias without access to weights or training data.
Iterative prompting with test feedback can cut the overall Code Bias Score for GPT-3.5-turbo-0125 from 60.58% to 8.77% while improving Pass@attribute from 66.60% to 85.66%.
Chain-of-Thought and positive role-play prompts alone do not reliably reduce bias and can even increase it, so naive 'be fair' prompting is not a sufficient mitigation strategy.
Temperature changes alter the Code Bias Score unevenly across models, so reported bias scores should be accompanied by the sampling configuration used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's demographic-parity definition implies that Solar would flag legitimate use of a protected attribute as bias; a natural extension is to separate 'protected attribute used as the genuine criterion' from 'protected attribute used as a proxy' based on the task's stated eligibility rule.
Because Solar is black-box and prompt-only, the same iteration loop could be adapted to other code-side harms, such as security vulnerabilities or privacy leaks, by replacing the fairness oracle with a security or privacy oracle.
The large improvement from iterative feedback suggests that much of the bias is elicited by the prompt rather than deeply fixed in the model weights; testing whether the residual 8.77% can be pushed toward zero with more iterations would probe that hypothesis.
The Pass@attribute metric measures attribute usage, not behavioral correctness; combining it with assertion-based functional tests would give a stronger guarantee that bias removal does not silently break intended conditions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before citing the 60% CBS number. The benchmark and framework are real contributions, but the headline severity and the 90% mitigation result both rest on a label mapping that isn't reported in the paper. An audit of that mapping is needed before the severity claim is taken at face value.

What's actually new: SocialBias-Bench gives 343 real-world human-centered tasks across seven categories, which goes beyond the contrived method-signature prompts in Liu et al. and the one-sentence prompts in Huang et al. Solar's DSL-based test generation is a reasonable black-box approach, and the BLS metric for bias direction is a useful addition. The comparison of mitigation strategies—CoT, role-playing, iterative prompting—is informative, and the finding that dialogue with Solar reduces bias while keeping functional correctness (as they measure it) is the kind of practical result practitioners want. The code and data are public, which is good.

Where it's soft: The construct validity of Eq. (1) is the load-bearing issue. It defines any outcome difference when a protected attribute changes as bias, with no allowance for legitimate use. The authors claim to handle this by reclassifying relevant demographics as related attributes, but they never show the per-task mapping. The stress-test worry is not hypothetical: three of the seven task categories (social benefits, health exams, licenses) plausibly require age, employment status, or marital status. Without the mapping, we can't tell whether a senior health screening task correctly treats age as related or incorrectly flags it as bias. The high age CBS (31.25%) and employment status (33.24%) for GPT-3.5 are exactly where false positives would concentrate. The methodology is not obviously wrong; it's just insufficiently documented.

Second, Pass@attribute is gameable. It counts sensitive attributes not used in the code as true negatives. So a model that simply drops all sensitive attributes gets a high score. Iterative prompting explicitly tells the model to avoid sensitive attributes, so the improvement from 66.60% to 85.66% is partially an artifact of the metric. This needs to be validated against a real functional correctness check.

Minor issues: only GPT-3.5 results are shown in the main table, there are no confidence intervals around CBS point estimates, and the 'up to 90%' claim doesn't match the 60.58% to 8.77% drop for GPT-3.5 (about 85%). These are minor by comparison.

Who it's for: SE and fairness researchers, and anyone deploying code LLMs in human-centered domains. The dataset and framework have value even if the severity conclusion is overstated. It deserves a serious referee, but the revision needs to release the sensitive/related mapping, fix Pass@attribute, and tone down the severity claim accordingly.

Recommendation: engage with it, but treat the severity numbers as provisional until the mapping is audited and the metric is tightened.

Referee Report

3 major / 6 minor

Summary. The paper introduces Solar, a black-box metamorphic-testing framework for detecting and mitigating social bias in LLM-generated code. Solar parses task definitions into code prompts and executable test cases that mutate one protected attribute at a time, and it reports Code Bias Score (CBS), Bias Leaning Score (BLS), and a proposed Pass@attribute correctness metric. The authors contribute SocialBias-Bench, a dataset of 343 human-centered coding tasks across seven categories, evaluate four LLMs (GPT-3.5-turbo-0125, codechat-bison@002, CodeLlama-70b-instruct-hf, claude-3-haiku-20240307), report severe bias (e.g., 60.58% CBS for GPT-3.5), and find that iterative prompting with Solar feedback substantially reduces CBS (e.g., to 8.77% for GPT-3.5) while Pass@attribute improves.

Significance. If the measurement framework is valid, this is a useful step toward automated, black-box fairness testing for code generation. Strengths include the publicly released artifacts, an extensible dataset with 343 tasks, the use of metamorphic testing as a concrete oracle mechanism, and the comparison of four LLMs under identical prompting conditions. The paper also goes beyond detection by exploring mitigation strategies and reporting statistical significance for the mitigation effects. The central quantitative claims, however, depend on the construct validity of the bias definition and on the independence of the evaluation oracle; the manuscript currently leaves both under-supported.

major comments (3)

[Methodology: Task generation and Eq. (1)] Equation (1) defines bias as any output difference caused by changing a protected attribute value. This is only a valid definition if protected attributes are never legitimate inputs for the task. The methodology states that when a demographic dimension is related to a task it is moved into the 'related attributes' set, but the paper does not report this reclassification for the 343 tasks, nor does it publish the resulting sensitive/related label mapping. Table 2 even states that all tasks in a category share the same related attributes, which is implausible across scenarios such as childcare-assistant eligibility and unemployment benefits. This is load-bearing because the dataset includes social benefits, health exams/programs, and licenses, where age, employment status, or marital status can be legitimate eligibility criteria. The high age (31.25%) and employment-status (33.24%) CBS values for GPT-3.5 in Table 3 are exactly where false positives would occur. Please audit the per-task labels, release the mapping, and recompute or re-analyze CBS after separating legitimate from illegitimate attribute use.
[Testing Code Bias, Eq. (5)] Pass@attribute counts sensitive attributes not used in the code as true negatives, so a snippet that omits all sensitive attributes is rewarded rather than penalized. This metric therefore cannot support the paper's claim that mitigation improves functional correctness 'without sacrificing functional correctness' (Section 'Results of Bias Mitigation Strategies'). The observed Pass@attribute increase from 66.60% to 85.66% in Table 5 is consistent with the model learning to drop attributes, not with improved task-level correctness. Please add a functional correctness measure that is independent of attribute usage, or reword the correctness claims to match the metric actually used.
[Bias Mitigation Strategies and Table 5] The mitigation evaluation uses the same Solar oracle for both feedback and assessment: Solar generates the iterative-prompting feedback and also computes the CBS and Pass@attribute values reported as outcomes. The reported reduction from 60.58% to 8.77% for GPT-3.5 is therefore not an independent demonstration of bias reduction. Please validate the mitigation results with an external bias test set or human annotation, and provide the exact feedback prompts used in each iteration so that the reader can distinguish 'removing sensitive attributes from the decision rule' from 'making a fair decision on the basis of all available information.'

minor comments (6)

[Abstract] The final sentence, 'Last, we make the code and data publicly available is highly extensible to evaluate new social problems,' is ungrammatical and should be rewritten.
[Testing Code Bias, Eq. (3)] The definition of $N_{bias}$ in the BLS formula is ambiguous; the text says 'as indicated by $N_b$' but the denominator is written as $N_{bias}$. Please align the notation and define what counts as a 'prejudicial reference' in code.
[Table 5] The table caption and body use inconsistent spellings ('Pass@attr', 'Pass@attribte', 'CBSdemographic'); also, the abstract's 'up to 90%' reduction is not directly visible in Table 5, so please state which model and iteration produces exactly 90%.
[Figure 4] The description of temperature effects is incomplete; phrases such as 'CBS increased from (t= 0.4) for GPT-3.5-turbo-0125' lack the actual CBS values at the relevant temperatures.
[Evaluation Results] CBS point estimates in Table 3 are reported as single percentages for 1,715 generated snippets per model; please add confidence intervals or per-task variances so the reader can assess the precision of the cross-model ordering.
[Conclusion / Related Work] The claim that Solar's bias testing 'does not have false positive detection' should be qualified: false positives can still arise from the label definitions in Eq. (1), even if the test execution is deterministic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the bias and mitigation results are empirical measurements under a stipulated fairness definition.

full rationale

The paper is an empirical measurement study rather than a derivation chain. Equation (1) stipulates a demographic-parity definition of bias, and CBS, BLS, and Pass@attribute are then defined metrics applied to observed LLM outputs. The conclusion that all four subject LLMs exhibit bias is a direct report of those measurements, not a result derived from the definition alone. No parameter is fitted and later renamed as a prediction, and no load-bearing result is imported from the authors' own prior work; the reference list contains no self-citation by the four authors. The iterative-prompting mitigation does reuse Solar as both the feedback generator and the evaluator, so the reported 90% reduction is demonstrated only with respect to Solar's own oracle and task labels; this is a same-oracle evaluation limitation rather than circularity, because the outcome depends on whether the LLMs actually comply with the feedback and could have failed. Similarly, the sensitive/related attribute labeling and the demographic-parity assumption are substantive validity threats, but they are assumptions, not circular steps. No specific circularity is identifiable in the paper's equations or reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central measurement rests on the authors' definitions of bias and functional correctness, plus the Solar-generated test oracle. No numerical parameters are fitted, but the construct validity of the bias metric and the custom correctness metric are load-bearing.

assumptions (4)

domain assumption Social bias in code is defined by demographic parity: any differential output when a single protected attribute changes is bias (Equation 1).
This definition treats all protected attribute differences as illegitimate, without considering contexts where attributes such as age or employment status may be legitimately relevant (e.g., health programs, senior benefits). Cited in Preliminaries and Methodology.
domain assumption The seven demographic dimensions are sensitive for all 343 tasks, and each task's ground-truth related attributes are complete and correct.
Task definitions were generated by GPT-4o and manually cross-checked by one author; no external validation. This grounds Pass@attribute and the test oracle.
domain assumption Pass@attribute, measuring use of related and sensitive attributes, is a valid proxy for functional correctness in social decision tasks.
The paper acknowledges leniency in social problems and uses this coarse-grained metric instead of exact output correctness; it rewards not using sensitive attributes (TN), which can inflate scores after mitigation.
domain assumption Generated code is executed in a harness where Solar's automatically generated test cases correctly instantiate the task and do not introduce bias or errors.
Solar relies on textX DSL to create Person classes and test cases; if the harness mis-specifies attribute types or values, measured differences could be artifacts.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bias Unveiled: Investigating Social Bias in LLM-Generated Code." pith.science (2026). https://pith.science/paper/IK3QVWNJ

@misc{pith2026241110351,
  author       = {Pith},
  title        = {Pith review of: Bias Unveiled: Investigating Social Bias in LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IK3QVWNJ}},
  note         = {Machine review of arXiv:2411.10351}
}

read the original abstract

Large language models (LLMs) have significantly advanced the field of automated code generation. However, a notable research gap exists in evaluating social biases that may be present in the code produced by LLMs. To solve this issue, we propose a novel fairness framework, i.e., Solar, to assess and mitigate the social biases of LLM-generated code. Specifically, Solar can automatically generate test cases for quantitatively uncovering social biases of the auto-generated code by LLMs. To quantify the severity of social biases in generated code, we develop a dataset that covers a diverse set of social problems. We applied Solar and the crafted dataset to four state-of-the-art LLMs for code generation. Our evaluation reveals severe bias in the LLM-generated code from all the subject LLMs. Furthermore, we explore several prompting strategies for mitigating bias, including Chain-of-Thought (CoT) prompting, combining positive role-playing with CoT prompting and dialogue with Solar. Our experiments show that dialogue with Solar can effectively reduce social bias in LLM-generated code by up to 90%. Last, we make the code and data publicly available is highly extensible to evaluate new social problems.

Figures

Figures reproduced from arXiv: 2411.10351 by the authors.

**Figure 2.** shows an example test case generated by Solar, it creates three instances of the Person class with specific attributes (i.e., alternating gender attributes and identical remaining attributes) and passes these attributes to the class constructor to create instances p1, p2, and p3 with these values. Next, it calls the suitable_for_journalist() method on the three instances. Last, the test case checks if the return va… view at source ↗

**Figure 3.** Radar chart: shape the pattern of prejudicial pref [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Illustration on the effect of hyper-parameters tem [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
cs.SE 2026-08 conditional novelty 6.0 of 10

On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Anthropic. 2024. Claude Models. https://docs.anthropic.com/en/docs/about-claude/models. Accessed: 2024-06-20

work page 2024
[4]

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021
[5]

Bai, Y.; Zhao, J.; Shi, J.; Wei, T.; Wu, X.; and He, L. 2023. FairBench: A Four-Stage Automatic Framework for Detecting Stereotypes and Biases in Large Language Models. arXiv preprint arXiv:2308.10397

arXiv 2023
[6]

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.; Tillet, P.; Such, F. P.; Cummings, D.; Plappert, M.; Ch...

arXiv 2021
[7]

Y.; Cheung, S

Chen, T. Y.; Cheung, S. C.; and Yiu, S. M. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543

arXiv 2020
[8]

M.; Sarro, F.; and Harman, M

Chen, Z.; Zhang, J. M.; Sarro, F.; and Harman, M. 2024. Fairness Improvement with Multiple Protected Attributes: How Far Are We? In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24

work page 2024

Show all 39 references

[9]

Corbett-Davies, S.; Pierson, E.; Feller, A.; Goel, S.; and Huq, A. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, 797--806

2017
[10]

Dejanovi \'c , I.; Vaderna, R.; Milosavljevi \'c , G.; and Vukovi \'c , Z . 2017. Textx: a python tool for domain-specific languages implementation. Knowledge-based systems, 115: 1--4

2017
[11]

Dhamala, J.; Sun, T.; Kumar, V.; Krishna, S.; Pruksachatkun, Y.; Chang, K.-W.; and Gupta, R. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 862--872

2021
[12]

M.; and Gergle, D

D \' az, M.; Johnson, I.; Lazar, A.; Piper, A. M.; and Gergle, D. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human factors in computing systems, 1--14

2018
[13]

Galhotra, S.; Brun, Y.; and Meliou, A. 2017. Fairness testing: testing software for discrimination. In Proceedings of the 2017 11th Joint meeting on foundations of software engineering, 498--510

2017
[14]

O.; Rossi, R

Gallegos, I. O.; Rossi, R. A.; Barrow, J.; Tanjim, M. M.; Kim, S.; Dernoncourt, F.; Yu, T.; Zhang, R.; and Ahmed, N. K. 2023. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770

2023 arXiv
[15]

Google. 2023. Code Chat. https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/code-chat. Accessed: 2024-06-20

2023
[16]

Huang, D.; Bu, Q.; Zhang, J.; Xie, X.; Chen, J.; and Cui, H. 2023. Bias Testing and Mitigation in LLM-based Code Generation. https://api.semanticscholar.org/CorpusID:262824773

2023
[17]

B.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Li, J.; Chim, J.; et al

Li, R.; Allal, L. B.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Li, J.; Chim, J.; et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161

2023 arXiv
[18]

P.; Wu, C.; Morency, L.-P.; and Salakhutdinov, R

Liang, P. P.; Wu, C.; Morency, L.-P.; and Salakhutdinov, R. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, 6565--6576. PMLR

2021
[19]

Liu, H.; Dacon, J.; Fan, W.; Liu, H.; Liu, Z.; and Tang, J. 2019. Does gender matter? towards fairness in dialogue systems. arXiv preprint arXiv:1910.10486

2019 arXiv
[20]

Liu, Y.; Chen, X.; Gao, Y.; Su, Z.; Zhang, F.; Zan, D.; Lou, J.-G.; Chen, P.-Y.; and Ho, T.-Y. 2023. Uncovering and quantifying social biases in code generation. Advances in Neural Information Processing Systems, 36

2023
[21]

Meade, N.; Poole-Dayan, E.; and Reddy, S. 2021. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. arXiv preprint arXiv:2110.08527

2021 arXiv
[22]

M e chura, M. 2022. A taxonomy of bias-causing ambiguities in machine translation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), 168--173

2022
[23]

Meta. 2024. Code Llama 70B Instruct HF. https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf. Accessed: 2024-06-20

2024
[24]

Mozafari, M.; Farahbakhsh, R.; and Crespi, N. 2020. Hate speech detection and racial bias mitigation in social media based on BERT model. PloS one, 15(8): e0237861

2020
[25]

Nadeem, M.; Bethke, A.; and Reddy, S. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456

2020 arXiv
[26]

Nangia, N.; Vania, C.; Bhalerao, R.; and Bowman, S. R. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133

2020 arXiv
[27]

Nijkamp, E.; Pang, B.; Hayashi, H.; Tu, L.; Wang, H.; Zhou, Y.; Savarese, S.; and Xiong, C. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474

2022 arXiv
[28]

OpenAI. 2022. GPT-3.5 Turbo Models. https://platform.openai.com/docs/models/gpt-3-5-turbo. Accessed: 2024-06-20

2022
[29]

M.; and Bowman, S

Parrish, A.; Chen, A.; Nangia, N.; Padmakumar, V.; Phang, J.; Thompson, J.; Htut, P. M.; and Bowman, S. R. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193

2021 arXiv
[30]

Rekabsaz, N.; and Schedl, M. 2020. Do neural ranking models intensify gender bias? In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2065--2068

2020
[31]

E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al

Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X. E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

2023 arXiv
[32]

Sap, M.; Card, D.; Gabriel, S.; Choi, Y.; and Smith, N. A. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, 1668--1678

2019
[33]

Sheng, E.; Chang, K.-W.; Natarajan, P.; and Peng, N. 2020. Towards controllable biases in language generation. arXiv preprint arXiv:2005.00268

2020 arXiv
[34]

Steed, R.; Panda, S.; Kobren, A.; and Wick, M. 2022. Upstream mitigation is not all you need: Testing the bias transfer hypothesis in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

2022
[35]

Wan, Y.; Wang, W.; He, P.; Gu, J.; Bai, H.; and Lyu, M. R. 2023. Biasasker: Measuring the bias in conversational ai system. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 515--527

2023
[36]

Wikipedia. 2024. Student's t-test. https://en.wikipedia.org/wiki/Student Accessed: 2024-06-20

2024
[37]

Yang, Z.; Yi, X.; Li, P.; Liu, Y.; and Xie, X. 2022. Unified detoxifying and debiasing in language generation via inference-time adaptive optimization. arXiv preprint arXiv:2210.04492

2022 arXiv
[38]

Zhang, M.; Sun, J.; Wang, J.; and Sun, B. 2023. TESTSGD: Interpretable testing of neural networks against subtle group discrimination. ACM Transactions on Software Engineering and Methodology, 32(6): 1--24

2023
[39]

Zhao, J.; Fang, M.; Pan, S.; Yin, W.; and Pechenizkiy, M. 2023. GPTBIAS: A comprehensive framework for evaluating bias in large language models. arXiv preprint arXiv:2312.06315

2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Anthropic. 2024. Claude Models. https://docs.anthropic.com/en/docs/about-claude/models. Accessed: 2024-06-20

work page 2024

[4] [4]

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

arXiv 2021

[5] [5]

Bai, Y.; Zhao, J.; Shi, J.; Wei, T.; Wu, X.; and He, L. 2023. FairBench: A Four-Stage Automatic Framework for Detecting Stereotypes and Biases in Large Language Models. arXiv preprint arXiv:2308.10397

arXiv 2023

[6] [6]

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.; Tillet, P.; Such, F. P.; Cummings, D.; Plappert, M.; Ch...

arXiv 2021

[7] [7]

Y.; Cheung, S

Chen, T. Y.; Cheung, S. C.; and Yiu, S. M. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543

arXiv 2020

[8] [8]

M.; Sarro, F.; and Harman, M

Chen, Z.; Zhang, J. M.; Sarro, F.; and Harman, M. 2024. Fairness Improvement with Multiple Protected Attributes: How Far Are We? In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24

work page 2024