REVIEW 4 cited by
FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks
read the original abstract
Large language models (LLMs) are increasingly used in high-stakes decisions such as hiring and college admissions, making their social bias a critical concern. While LLMs are trained to refuse explicitly biased requests, bias can be leaked implicitly during LLM planning and reasoning process. As code becomes the primary medium for LLM internal logic-writing, we introduce FairCoder, a benchmark that frames decision-making as coding tasks to systematically probe LLM bias across employment, education, and healthcare domains, covering multiple fairness definitions. Considering that existing metrics may fail when LLMs frequently refuse the request, we propose FairScore, a metric that jointly captures refusal behavior and group-level outcome diversity. Experiments with a 1k-sample dataset on powerful LLMs reveal consistent and previously underexplored bias patterns, such as prioritizing applicants from high-income families in college admissions. Our findings highlight the risks of deploying LLMs as decision-making agents and provide a comprehensive evaluation framework for future research.
Forward citations
Cited by 4 Pith papers
-
From Codebases to LLMs: Non-Inclusive Naming in Linux Foundation Repositories
Non-inclusive INI terms in Linux Foundation repos declined ~47% since 2020 but remain in 62.7% of codebases, and LLMs still reconstruct them from context.
-
Biased or Personalized? The Impact of Personal Information on AI-driven Development
Changing only the prompter's age and gender in AI coding prompts produces statistically significant differences in generated website interface design, template content, and code structure across 800 generated websites...
-
From Codebases to LLMs: Non-Inclusive Naming in Linux Foundation Repositories
Non-inclusive INI terms in LF repos fell ~47% since 2020 but remain in 62.7% of identifier sets and are often reconstructed by LLMs from surrounding code.
-
Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.