REVIEW 14 cited by
Evaluating and Mitigating Discrimination in Language Model Decisions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Evaluating and Mitigating Discrimination in Language Model Decisions
read the original abstract
As language models (LMs) advance, interest is growing in applying them to high-stakes societal decisions, such as determining financing or housing eligibility. However, their potential for discrimination in such contexts raises ethical concerns, motivating the need for better methods to evaluate these risks. We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed. Specifically, we use an LM to generate a wide array of potential prompts that decision-makers may input into an LM, spanning 70 diverse decision scenarios across society, and systematically vary the demographic information in each prompt. Applying this methodology reveals patterns of both positive and negative discrimination in the Claude 2.0 model in select settings when no interventions are applied. While we do not endorse or permit the use of language models to make automated decisions for the high-risk use cases we study, we demonstrate techniques to significantly decrease both positive and negative discrimination through careful prompt engineering, providing pathways toward safer deployment in use cases where they may be appropriate. Our work enables developers and policymakers to anticipate, measure, and address discrimination as language model capabilities and applications continue to expand. We release our dataset and prompts at https://huggingface.co/datasets/Anthropic/discrim-eval
Forward citations
Cited by 14 Pith papers
-
Reference Feature Atlases for Mechanistic Auditing of Language Models
A reference feature atlas attaches a new language model with a linear decoder and surfaces panel-uncovered structure via a separate residual dictionary.
-
FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
Audit format (rate vs rank vs allocate; transparent vs disguised) reverses the apparent direction of LLM demographic bias, while causal framing of need dominates allocations by roughly an order of magnitude.
-
Defeat Devices in AI Systems
The paper defines defeat devices in AI via a triadic test (discriminator, concealed swap, performance gap), unifies existing cases under this concept, proposes TADP detection, and claims such devices can emerge natura...
-
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
A standardized evaluation framework shows comparative bias tests in LLMs activate more latent discrimination than isolated assessments, worsened by CoT and scaling with model size.
-
AgentFairBench: Do LLM Agents Discriminate When They Act?
AgentFairBench is a multi-domain benchmark for demographic disparity in LLM agent actions, with a pilot showing no significant effect for Claude Haiku 4.5 after arity-matched noise correction.
-
What Do People Actually Want From AI? Mapping Preference Plurality
Open-ended preference data reveals substantial plurality in what people want from AI and divergent interpretations of shared values such as truthfulness.
-
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
Standardized-test benchmarks for LLM fairness are unreliable because prompt wording alone drives most score variance and ranking changes, while a multi-agent conversational framework reveals consistent model-specific ...
-
Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest
Many LLMs prioritize company ad incentives over user welfare by recommending pricier sponsored products, disrupting purchases, or concealing prices in comparisons.
-
DeFrame: Debiasing Large Language Models Against Framing Effects
LLM fairness scores shift substantially with positive vs negative framing of the same question, and DeFrame—a three-step self-revision prompt—reduces both average bias and this framing gap.
-
White-Box Sensitivity Auditing with Steering Vectors
White-box steering along gender/race concept vectors reveals larger and more consistent bias signals than black-box prompt perturbation in simulated LLM decision audits.
-
White-Box Sensitivity Auditing with Steering Vectors
A white-box sensitivity auditing framework using activation steering detects substantial dependence on protected attributes in LLM predictions on simulated decision tasks, even when black-box evaluations indicate little bias.
-
Laissez-Faire Harms: Algorithmic Biases in Generative Language Models
Generative LMs in laissez-faire open-ended prompting settings disproportionately generate subordinated portrayals of minoritized race, gender, and sexual orientation identities at rates hundreds to thousands of times ...
-
Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows
Changing the file format of identical input content changes LLM workflow decisions in 41% of cases on average and can reduce accuracy by up to 56 percentage points, with CSV the most error-prone format.
-
OpenAI o1 System Card
OpenAI reports that chain-of-thought reasoning in o1 models enables deliberative alignment, yielding state-of-the-art results on selected safety benchmarks for illicit advice, stereotypes, and jailbreaks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.