Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a language model's bias shows up as asymmetric alignment of concept vectors inside the model, measurable without any labeled test set.

desk verdict BiasLens is a fresh idea with a flawed validation: the paper's own Table 1 contradicts the r>0.85 claim, and the concept vector extraction is likely polluted by steering and dataset-format artifacts. read the letter →

arxiv 2505.15524 v1 pith:O5G3J4GM submitted 2025-05-21 cs.CL cs.AI

Lang Gao , Kaiyang Wan , Wei Liu , Chenxi Wang , Zirui Song , Zixiang Xu , Yanbo Wang , Veselin Stoyanov
show 1 more author
Xiuying Chen
This is my paper
classification cs.CLcs.AI
keywords biasevaluationconceptactivationvectorssparseautoencoderstest-set-freerepresentationgeometryLLMfairnessinterpretabilitydiscovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that bias in large language models can be measured from the geometry of the model's own concept representations, with no manually labeled test set. It defines bias as the asymmetry between a target concept's alignment with two reference concepts, and it introduces BiasLens, which extracts each concept as a vector by steering the model along concept activation vectors and projecting the activation change through a sparse autoencoder. The paper reports that BiasLens agrees with traditional behavioral bias metrics, with Spearman correlations above 0.85 for its central comparisons and near 0.94 for the F1-difference metric on both Gemma models tested. It also claims to surface biases that are hard to detect behaviorally, such as a patient's insurance status shifting the model's diagnostic associations. If these claims hold, bias evaluation becomes automatic, scalable, and applicable to any concept a user can name.

What carries the argument

The central object is the concept representation vector $\vec{C} = \mathrm{Norm}(z_{\mathrm{steer}}) - \mathrm{Norm}(z_{\mathrm{ori}})$, where $z_{\mathrm{ori}}$ and $z_{\mathrm{steer}}$ are sparse autoencoder encodings of the final-layer activation before and after CAV-based steering toward the concept. This vector is meant to isolate the dimensions a concept activates, in a space whose features are individually interpretable. The bias score for a target $t$ against references $r_1$ and $r_2$ is $S_{\mathrm{bias}} = \left|\cos\angle(\vec{C}_t,\vec{C}_{r_1}) - \cos\angle(\vec{C}_t,\vec{C}_{r_2})\right|$, and the paper's validation shows that normalizing and subtracting the two encodings raises the prominence of concept-relevant SAE features in the resulting vector.

What would settle it

Replace the concept activation vectors with random unit vectors of the same norm, keep the steering threshold and sparse autoencoder projection unchanged, and compare the resulting bias scores; if they do not fall to near zero, the method is measuring generic activation drift rather than concept alignment.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that a model's bias is carried by the relative directions of concept vectors: when the vector for a target concept such as 'surgery' lies closer to 'public insurance' than to 'private insurance', the model exhibits conceptual correlation bias even if no test prompt reveals it in behavior. The method operationalizes this by training per-layer linear classifiers to obtain concept activation vectors, iteratively steering activations toward each concept until a classifier reaches 99.9 percent confidence, encoding the before and after final-layer activations with a sparse autoencoder, and taking the normalized difference as the concept vector. Bias is then the absolute difference in cosine similarity between the target vector and the two reference vectors. The paper argues that these representation-level scores track behavioral bias metrics and can be computed for arbitrary concepts, which lets the method expose biases for which no test set exists.

Load-bearing premise

The method assumes that the difference between the steered and unsteered activations, after sparse autoencoder projection, reflects only the target concept rather than arbitrary or unrelated changes introduced by the steering process.

Editorial extensions

If this is right

  • Bias evaluation can run fully automatically once a user names a target concept and a reference pair, so new bias axes need not wait for annotated datasets.
  • Under-served domains such as clinical and educational settings become auditable for concepts no existing benchmark covers, such as insurance status in diagnosis.
  • The representation-space scores rank biases similarly to behavioral metrics such as F1-diff and SEAT, so they can substitute for or complement those metrics when test sets are unavailable.
  • The observed roughly 50x speedup over manual annotation pipelines means large-scale sweeps across many concept pairs are feasible.
  • Because the method surfaces concept asymmetries, it can serve as a discovery tool for hypotheses that later behavioral tests confirm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the correlation with behavioral metrics holds across more models and concepts, BiasLens could be used to audit models before deployment, flagging concept asymmetries that later translate into behavioral discrimination; the paper does not itself test this predictive direction.
  • The reliance on a single steering prompt (acknowledged in the paper's limitations) suggests the score may be prompt-sensitive; averaging over diverse prompts is a natural extension that could turn the method into a stable estimator.
  • The method implicitly assumes that cosine similarity is the right measure of concept alignment; replacing it with kernel or geometric distance measures could change which asymmetries are considered biases, and the paper leaves that comparison open.
  • Because SAE features are shared across concepts, the extracted vectors could be decomposed into feature-level contributions, potentially allowing bias to be attributed to specific interpretable features — an extension the paper does not pursue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes BiasLens, a test-set-free bias evaluation framework for LLMs. Bias is formalized as asymmetric alignment (Eq. 2, Eq. 4) between a target concept vector and two reference concept vectors in SAE space. Concept vectors are built by training CAVs on GPT-4o-generated positive sentences versus OpenWebText negatives (Section 3.2.1), steering the model at every layer until classifier confidence exceeds 0.999 (Algorithm 1), and taking the normalized difference of SAE-encoded activations before and after steering (Section 3.2.2). The method is evaluated on three LLMs by Spearman correlation with four extrinsic and two intrinsic behavioral bias metrics (Section 4.2), and used to flag biases in medical and educational scenarios (Section 4.3). The abstract claims strong agreement with traditional metrics (r > 0.85) and discovery of hard-to-detect biases.

Significance. The motivation is timely: existing behavioral bias tests require human-curated data and cover only a limited set of concepts. If the geometric representation were validated to be concept-specific, BiasLens would be a scalable, interpretable, and low-cost complement to behavior-based evaluation, and the code/data release supports reproducibility. The method is clearly described, uses publicly available SAEs, and the single-concept analysis in Figure 3 is instructive. However, the central empirical claims are not currently supported by the evidence in the manuscript. The claimed r > 0.85 is contradicted by Table 1; the validation is based on only six concepts with no uncertainty; and the key assumption that the vector C is specific to the injected concept is untested. These are correctness issues, not presentation issues; the paper currently overstates what it demonstrates.

major comments (4)
  1. [4.2, Table 1 and Figure 4] The abstract's claim of 'strong agreement with traditional bias evaluation metrics, Spearman correlation r > 0.85' is contradicted by the paper's own results. Table 1 shows BiasLens correlations ranging from 0.1531 (Perplexity, Llama 3.1 8B) to 0.9429, with negative values (EOD = -0.7143, I.F. = -0.7143 on Llama 3.1 8B); even the strongest extrinsic correlation on Llama is 0.7143 for |F1-Diff|. The text in Section 4.2 also claims correlations 'close to 1' on both Gemma models, but Table 1 shows only |F1-Diff| reaches 0.9429 while EOD is 0.1429 on Gemma 2 2B and 0.60 on Gemma 2 9B. The central validation claim needs to be restated or supported by additional experiments.
  2. [3.2.2, Algorithm 1, Eq. (4)] The semantic grounding of the extracted concept vector is assumed, not established. CAVs are trained to separate GPT-4o-generated concept sentences from OpenWebText sentences; these sets differ in style, format, and lexica, so the separating hyperplane v^(l) may encode prompt format or genre rather than the intended concept. The iterative steering to classifier confidence 0.999 (Algorithm 1) can push activations off the data manifold, and Figure 3(a) reports that 85.26% of activated features after steering are unrelated to the concept. Since the bias score in Eq. (4) is a cosine difference of vectors C obtained this way, any off-concept contamination propagates to all correlations and new-bias claims. This assumption needs explicit validation: for example, style-matched negative probes, random-direction or unrelated-concept controls, and invariance of final bias scores under probe-distribution changes. Appendix B.7 reports only classifier accuracy, which does not establish score robustness.
  3. [4.2] The statistical support for the main validation is weak. All Spearman correlations are computed over only six target concepts (Table 10), no error bars, confidence intervals, bootstrap, or repeated runs are reported, and Appendix A concedes that using a single steering prompt introduces variability. With n=6, a correlation of 0.94 is not statistically distinguishable from much smaller values, so the qualitative ranking across models may not be reliable. The authors should report permutation tests or bootstrap intervals and vary prompt and probe seeds.
  4. [4.3, Table 3] The claim that BiasLens 'reveals' previously hard-to-detect biases, such as insurance status affecting diagnostic assessments, is not supported by the presented evidence. The bias scores are raw cosine differences (e.g., 0.1134 for public versus private insurance on surgery) with no uncertainty, no comparison baseline, and no behavioral or human validation. Large and small scores are interpreted without any calibration or significance threshold. At minimum, the authors need to show that these effects are stable across prompts and probe sets and are larger than those produced by neutral control concepts.
minor comments (5)
  1. [Table 1] The Llama 3.1 8B row contains the entry '0.7143-0.9429', which is missing a separator; from Figure 4(c) it should read 0.7143 and -0.9429.
  2. [Appendix C.2] Appendix C.2 says the paper compares against 'eight widely used bias evaluation metrics' and C.2.1 says 'six group-based fairness metrics,' but only four extrinsic and two intrinsic metrics are actually used; the counts should be harmonized.
  3. [Appendix C.3.2] The phrase 'To calculate PG' introduces an undefined quantity; either define PG or remove the phrase.
  4. [Appendix B.7] The conclusion that BiasLens is 'robust to the construction and content of the probing dataset' is drawn solely from CAV classifier accuracy; as noted in Major Comment 2, the relevant quantity is the stability of bias scores, which is not reported.
  5. [Abstract and Section 1] The phrases 'even without labeled data' and 'fully automatic evaluation' should be qualified, since the CAV probe sentences are generated by GPT-4o and the SAE feature interpretations rely on Neuronpedia and GPT-4o-mini; the claim is test-set-free with respect to manual labels, not label-free in an absolute sense.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: the bias score is an independent geometric measure validated against external behavioral metrics; the only self-citation is peripheral.

full rationale

The paper's central claim is that BiasLens vector-space scores agree with behavioral bias metrics and reveal new biases. The bias score (Eq. 4) is computed from activations after CAV steering and SAE encoding; it is not fitted to the baseline metrics. The CAV is trained via logistic regression on GPT-4o-generated positive sentences versus OpenWebText negatives; this is a probe-data dependence, but the validation against Yelp/IMDB and WinoBias metrics uses separate corpora and labels, so the agreement is an empirical correlation, not a tautology. The assumption in Sec. 3.2.2 that steering affects only concept-relevant dimensions is an empirical hypothesis tested in Sec. 3.3 and Figure 3 via SAE feature annotations (Neuronpedia plus GPT-4o-mini), not a definition that entails the result. No equation reduces to another by construction in the paper's derivation chain. The only self-citation is Ref. [28] in related work, supporting a peripheral claim about LLM-as-judge output preferences; it is not load-bearing for any derivation or validation. Residual concerns about GPT-4o involvement in probe generation and feature labeling are limitations or dependence issues, not circular reductions: the central agreement claim is anchored by independent behavioral benchmarks with separate text corpora and labels. Therefore, no specific circular step meets the threshold of Eq. X = Eq. Y by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method depends on several modeling assumptions and hand-set hyperparameters that are not stress-tested. The most load-bearing are the faithfulness of the CAV steering plus SAE difference concept vector, and the adequacy of GPT-4o-generated probes.

free parameters (6)
  • Steering step size delta = 1
    Hand-chosen in Algorithm 1; affects how far activations are pushed along the CAV and therefore the resulting concept vector.
  • Steering confidence threshold tau = 0.999
    Stopping criterion in Algorithm 1; no sensitivity analysis is provided.
  • Probe dataset size = 150 positive, 150 negative sentences per concept
    Fixed number of GPT-4o-generated positive sentences and OpenWebText negatives used to train CAVs; no ablation on this size.
  • Single steering prompt per model and concept = One prompt selected empirically per experiment
    Appendix B.4 states prompts are 'empirically valid' and Appendix A acknowledges a single prompt may cause variability.
  • p-value filter for intrinsic baselines = p <= 0.05
    Only statistically significant SEAT and perplexity results are retained for correlation computation, which changes the sample and can inflate correlations.
  • SAE width and layer choice = 16k or 32k features on the last layer
    SAEs are selected from SAELens with similar dimensionality; no ablation across layers or widths is reported.
assumptions (6)
  • domain assumption Last-token activations at each layer capture the semantic meaning of the entire sentence.
    Invoked in Section 3.2.1 when extracting activations for CAV training; if false, the CAVs may not represent the intended concepts.
  • domain assumption The logistic regression weight vector points from general language toward the target concept representation.
    Standard TCAV assumption used in Section 3.2.1; the paper relies on it to justify steering along the CAV.
  • ad hoc to paper Steering until classifier confidence exceeds 0.999 yields a concept-steered model without producing out-of-distribution activations.
    The threshold and iterative steering procedure are introduced for this method and are not validated against distribution shift.
  • domain assumption The difference between SAE encodings before and after steering is dominated by concept-relevant features.
    This is the core extraction assumption in Section 3.2.2; if steering shifts many unrelated features, the concept vector is contaminated.
  • domain assumption Cosine similarity between concept vectors is an adequate measure of conceptual alignment.
    Used in the bias score definition, Eq. (4); Appendix A acknowledges this may fail for complex nonlinear relationships.
  • domain assumption GPT-4o-generated sentences are valid and neutral probes for the target concepts.
    CAV training relies on these synthetic positive examples; any bias or artifact in GPT-4o outputs is inherited by the concept vectors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs." pith.science (2026). https://pith.science/paper/O5G3J4GM

@misc{pith2026250515524,
  author       = {Pith},
  title        = {Pith review of: Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5G3J4GM}},
  note         = {Machine review of arXiv:2505.15524}
}
read the original abstract

Bias in Large Language Models (LLMs) significantly undermines their reliability and fairness. We focus on a common form of bias: when two reference concepts in the model's concept space, such as sentiment polarities (e.g., "positive" and "negative"), are asymmetrically correlated with a third, target concept, such as a reviewing aspect, the model exhibits unintended bias. For instance, the understanding of "food" should not skew toward any particular sentiment. Existing bias evaluation methods assess behavioral differences of LLMs by constructing labeled data for different social groups and measuring model responses across them, a process that requires substantial human effort and captures only a limited set of social concepts. To overcome these limitations, we propose BiasLens, a test-set-free bias analysis framework based on the structure of the model's vector space. BiasLens combines Concept Activation Vectors (CAVs) with Sparse Autoencoders (SAEs) to extract interpretable concept representations, and quantifies bias by measuring the variation in representational similarity between the target concept and each of the reference concepts. Even without labeled data, BiasLens shows strong agreement with traditional bias evaluation metrics (Spearman correlation r > 0.85). Moreover, BiasLens reveals forms of bias that are difficult to detect using existing methods. For example, in simulated clinical scenarios, a patient's insurance status can cause the LLM to produce biased diagnostic assessments. Overall, BiasLens offers a scalable, interpretable, and efficient paradigm for bias discovery, paving the way for improving fairness and transparency in LLMs.

Figures

Figures reproduced from arXiv: 2505.15524 by the authors.

Figure 1
Figure 1. Comparison between traditional behavior-based and our representation-based bias evalua [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of BIASLENS . A running example using the concept “doctor” illustrates the three main steps of our method: (1) CAV derivation: train linear classifiers at each layer using random and doctor-related sentences, and use the classifier weights as CAVs; (2) Concept representation extraction: extract model activations before and after steering with “doctor” CAVs, project them into SAE space, and subtract the norm… view at source ↗
Figure 3
Figure 3. Validation of concept representation extraction. (a) CAV-based steering activates relevant [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Spearman correlation matrices between BIASLENS and four extrinsic behavioral metrics on (a) Gemma 2 2B, (b) Gemma 2 9B, and (c) Llama 3.1 8B. Each matrix shows pairwise correlations computed over 6 target concepts. Intrinsic behavioral metrics. We compare BIASLENS with…
Figure 5
Figure 5. Figure 5: Prompt template for synthesizing positive probe samples using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for classifying SAE features as concept-relevant or not using GPT-4o-mini. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Case analysis of failed steering with successful concept extraction. (a) Concept-related [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Prompt template for sentiment classification task. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

    cs.LG 2026-07 conditional novelty 6.5 of 10

    LLM-as-judge scoring biases concentrate in low-dimensional, type-specific activation subspaces that support bidirectional causal steering and cross-domain failure prediction.

Reference graph

Works this paper leans on

83 extracted references · 57 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llms are biased teachers: Evaluating llm bias in personalized education.arXiv preprint arXiv:2410.14012, 2024

    Iain Weissburg, Sathvika Anand, Sharon Levy, and Haewon Jeong. Llms are biased teachers: Evaluating llm bias in personalized education.arXiv preprint arXiv:2410.14012, 2024

    arXiv 2024

  2. [2]

    Sociodemographic biases in medical decision making by large language models.Nature Medicine, pages 1–9, 2025

    Mahmud Omar, Shelly Soffer, Reem Agbareia, Nicola Luigi Bragazzi, Donald U Apakama, Carol R Horowitz, Alexander W Charney, Robert Freeman, Benjamin Kummer, Benjamin S Glicksberg, et al. Sociodemographic biases in medical decision making by large language models.Nature Medicine, pages 1–9, 2025

    2025

  3. [3]

    Gender bias and stereotypes in large language models

    Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12–24, 2023

    2023

  4. [4]

    Hannah Rose Kirk, Yennie Jun, Filippo V olpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models.Advances in neural information processing systems, 34:2611–2624, 2021

    2021

  5. [5]

    Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020

    Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020

    arXiv 2004

  6. [6]

    Gender bias in coreference resolution: Evaluation and debiasing methods

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2...

    work page 2018

  7. [7]

    Bowman, and Rachel Rudinger

    Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short...

    work page 2019

  8. [8]

    Bias and volatility: A statistical framework for evaluating large language model's stereotypes and the associated generation inconsistency

    Yiran Liu, Ke Yang, Zehan Qi, Xiao Liu, Yang Yu, and ChengXiang Zhai. Bias and volatility: A statistical framework for evaluating large language model's stereotypes and the associated generation inconsistency. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, vo...

    work page 2024

Show all 83 references
  1. [9]

    Climb: A benchmark of clinical bias in large language models, 2024

    Yubo Zhang, Shudi Hou, Mingyu Derek Ma, Wei Wang, Muhao Chen, and Jieyu Zhao. Climb: A benchmark of clinical bias in large language models, 2024

    2024

  2. [10]

    Hossein Azarpanah and Mohsen Farhadloo. Measuring biases of word embeddings: What similarity measures and descriptive statistics to use? In Yada Pruksachatkun, Anil Ramakrishna, Kai-Wei Chang, Satyapriya Krishna, Jwala Dhamala, Tanaya Guha, and Xiang Ren, editors,Proceedings o...

    2021

  3. [11]

    On measuring and mitigating biased inferences of word embeddings

    Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7659–7666, 2020

    2020

  4. [12]

    Cai, James Wexler, Fernanda B

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InProceedings of the 35th International Conference on Machine...

    2018

  5. [13]

    Controlling large language models through concept activation vectors.arXiv preprint arXiv:2501.05764, 2025

    Hanyu Zhang, Xiting Wang, Chengao Li, Xiang Ao, and Qing He. Controlling large language models through concept activation vectors.arXiv preprint arXiv:2501.05764, 2025

    2025 arXiv

  6. [14]

    Can sparse autoencoders be used to decompose and interpret steering vectors?CoRR, abs/2411.08790, 2024

    Harry Mayne, Yushi Yang, and Adam Mahdi. Can sparse autoencoders be used to decompose and interpret steering vectors?CoRR, abs/2411.08790, 2024. 10

    2024 arXiv

  7. [15]

    Sparse autoen- coders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [17]

    People’s perceptions toward bias and related concepts in large language models: A systematic review, 2024

    Lu Wang, Max Song, Rezvaneh Rezapour, Bum Chul Kwon, and Jina Huh-Yoo. People’s perceptions toward bias and related concepts in large language models: A systematic review, 2024

    2024

  10. [18]

    Bias and volatility: A statistical framework for evaluating large language model’s stereotypes and the associated generation inconsistency

    Yiran Liu, Ke Yang, Zehan Qi, Xiao Liu, Yang Yu, and Cheng Xiang Zhai. Bias and volatility: A statistical framework for evaluating large language model’s stereotypes and the associated generation inconsistency. Advances in Neural Information Processing Systems, 37:110131–110155, 2024

    2024

  11. [19]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P....

    2024

  12. [20]

    A survey on fairness in large language models.arXiv preprint arXiv:2308.10149, 2023

    Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models.arXiv preprint arXiv:2308.10149, 2023

    2023 arXiv

  13. [21]

    On measures of biases and harms in NLP

    Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. On measures of biases and harms in NLP. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, editors,Findings of the As...

    2022

  14. [22]

    Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value-laden questions? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15185–15221, Miami, Florida,...

    2024

  15. [23]

    Exploring value biases: How llms deviate towards the ideal, 2024

    Sarath Sivaprasad, Pramod Kaushik, Sahar Abdelnabi, and Mario Fritz. Exploring value biases: How llms deviate towards the ideal, 2024

    2024

  16. [24]

    Writing style matters: An examination of bias and fairness in information retrieval systems

    Hongliu Cao. Writing style matters: An examination of bias and fairness in information retrieval systems. InProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, page 336–344, New York, NY , USA, 2025. Association for Computing Machinery

    2025

  17. [25]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Measuring gender and racial biases in large language models.arXiv preprint arXiv:2403.15281, 2024

    Jiafu An, Difang Huang, Chen Lin, and Mingzhu Tai. Measuring gender and racial biases in large language models.arXiv preprint arXiv:2403.15281, 2024

    2024 arXiv

  19. [27]

    you gotta be a doctor, lin

    Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daumé Iii. “you gotta be a doctor, lin” : An investigation of name-based bias of large language models in employment recommendations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference ...

    2024

  20. [28]

    Justice or prejudice? quantifying biases in LLM-as-a-judge

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. InThe Thirteenth International Conference on Learning Re...

    2025

  21. [29]

    Large language models propagate race-based medicine.NPJ Digital Medicine, 6(1):195, 2023

    Jesutofunmi A Omiye, Jenna C Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou. Large language models propagate race-based medicine.NPJ Digital Medicine, 6(1):195, 2023

    2023

  22. [30]

    Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1):176, 2024

    Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, and Zhiyong Lu. Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1):176, 2024. 11

    2024

  23. [31]

    Racial differences in pain assessment and false beliefs about race in ai models.JAMA Network Open, 7(10):e2437977–e2437977, 2024

    Brototo Deb and Adam Rodman. Racial differences in pain assessment and false beliefs about race in ai models.JAMA Network Open, 7(10):e2437977–e2437977, 2024

    2024

  24. [32]

    Bowen III, S

    Donald E. Bowen III, S. McKay Price, Luke C.D. Stein, and Ke Yang. Measuring and mitigating racial disparities in large language model mortgage underwriting. http://dx.doi.org/10.2139/ssrn. 4812158, April 2024. Available at SSRN:https://ssrn.com/abstract=4812158

    2024 doi

  25. [33]

    Rahul Vats, Shekhar Agrawal, and Srinivasa Chippada. Bias detection and fairness in large language models for financial services.International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11:1329–1345, 03 2025

    2025

  26. [34]

    Evaluating large language models: A comprehensive survey, 2023

    Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. Evaluating large language models: A comprehensive survey, 2023

    2023

  27. [35]

    StereoSet: Measuring stereotypical bias in pretrained language models

    Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the ...

  28. [36]

    Unmasking the mask–evaluating social biases in masked language models

    Masahiro Kaneko and Danushka Bollegala. Unmasking the mask–evaluating social biases in masked language models. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 11954–11962, 2022

    2022

  29. [37]

    Measuring bias in contex- tualized word representations

    Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contex- tualized word representations. In Marta R. Costa-jussà, Christian Hardmeier, Will Radford, and Kellie Webster, editors,Proceedings of the First Workshop on Gender Bias in Natura...

    2019

  30. [38]

    On measuring social biases in sentence encoders.arXiv preprint arXiv:1903.10561, 2019

    Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders.arXiv preprint arXiv:1903.10561, 2019

    1903 arXiv

  31. [39]

    Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases

    Wei Guo and Aylin Caliskan. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 122–133, New York, NY , USA, 2021. Associa...

    2021

  32. [40]

    Semantics derived automatically from language corpora contain human-like biases.Science, 356(6334):183–186, 2017

    Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases.Science, 356(6334):183–186, 2017

    2017

  33. [41]

    Understanding the origins of bias in word embeddings

    Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. Understanding the origins of bias in word embeddings. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 ofP...

    2019

  34. [42]

    Explaining explainability: Recommendations for effective use of concept activation vectors.Transactions on Machine Learning Research, 2025

    Angus Nicolson, Lisa Schut, Alison Noble, and Yarin Gal. Explaining explainability: Recommendations for effective use of concept activation vectors.Transactions on Machine Learning Research, 2025

    2025

  35. [43]

    Uncovering safety risks of large language models through concept activation vector

    Zhihao Xu, Ruixuan HUANG, Changyu Chen, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  36. [44]

    Controlling large language models through concept activation vectors, 2025

    Hanyu Zhang, Xiting Wang, Chengao Li, Xiang Ao, and Qing He. Controlling large language models through concept activation vectors, 2025

    2025

  37. [45]

    Steering llama 2 via contrastive activation addition, 2024

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024

    2024

  38. [46]

    Steering llms’ behavior with concept activation vectors, September 2024

    Ruixuan Huang. Steering llms’ behavior with concept activation vectors, September 2024. Draft manuscript. Available on LessWrong forum

    2024

  39. [47]

    Extracting unlearned information from llms with activation steering, 2024

    Atakan Seyito˘glu, Aleksei Kuvshinov, Leo Schwinn, and Stephan Günnemann. Extracting unlearned information from llms with activation steering, 2024

    2024

  40. [48]

    Sparse autoencoder

    Andrew Ng. Sparse autoencoder. https://web.stanford.edu/class/cs294a/ sparseAutoencoder_2011new.pdf, 2011. CS294A Lecture Notes, Stanford University

    2011

  41. [49]

    Efficient training of sparse autoencoders for large language models via layer groups.arXiv preprint arXiv:2410.21508, 2024

    Davide Ghilardi, Federico Belotti, and Marco Molinari. Efficient training of sparse autoencoders for large language models via layer groups.arXiv preprint arXiv:2410.21508, 2024. 12

    2024

  42. [50]

    Efficient dictionary learning with switch sparse autoencoders.arXiv preprint arXiv:2410.08201, 2024

    Anish Mudide, Joshua Engels, Eric J Michaud, Max Tegmark, and Christian Schroeder de Witt. Efficient dictionary learning with switch sparse autoencoders.arXiv preprint arXiv:2410.08201, 2024

    2024 arXiv

  43. [51]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.arXiv preprint arXiv:2407.14435, 2024

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.arXiv preprint arXiv:2407.14435, 2024

    2024 arXiv

  44. [52]

    Smith and Jonas Brinkmann

    Luke R. Smith and Jonas Brinkmann. Interpreting preference models with sparse autoencoders.AI Alignment Forum, 2024

    2024

  45. [53]

    Effectiveness of sparse autoencoder for understanding and removing gender bias in LLMs

    Praveen Hegde. Effectiveness of sparse autoencoder for understanding and removing gender bias in LLMs. InNeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024

    2024

  46. [54]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  47. [55]

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings.Advances in neural information processing systems, 29, 2016

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings.Advances in neural information processing systems, 29, 2016

    2016

  48. [56]

    The woman worked as a babysitter: On biases in language generation

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language ...

    2019

  49. [57]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024

  50. [58]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  51. [59]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  52. [60]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  53. [61]

    k-sparse autoencoders, 2014

    Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2014

    2014

  54. [62]

    Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023

    Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. Software available from neuronpedia.org

    2023

  55. [63]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  56. [64]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  57. [65]

    Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. Quantifying social biases in NLP: A generalization and empirical comparison of extrinsic fairness metrics.Transactions of the Association for Computational Linguistics, 9:1249–1267, 2021

    2021

  58. [66]

    Zhang, Federica Sarro, and Mark Harman

    Max Hort, Jie M. Zhang, Federica Sarro, and Mark Harman. Search-based automatic repair for fairness and accuracy in decision-making software.Empirical Software Engineering, 29(1):36, 2024

    2024

  59. [67]

    Reducing sentiment bias in language models via counterfactual evaluation

    Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Associati...

    2020

  60. [68]

    A survey on fairness in large language models, 2024

    Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models, 2024

    2024

  61. [69]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015. 14

    2015

  62. [70]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors,Proceedings of the 49th Annual Meeting of the Association for Computational ...

    2011

  63. [71]

    Explore spurious correlations at the concept level in language models for text classification

    Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. Explore spurious correlations at the concept level in language models for text classification. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  64. [72]

    RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models

    Soumya Barikeri, Anne Lauscher, Ivan Vuli´c, and Goran Glavaš. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of t...

    1941

  65. [73]

    Towards detecting unanticipated bias in large language models, 2024

    Anna Kruspe. Towards detecting unanticipated bias in large language models, 2024

    2024

  66. [74]

    Edu-values: Towards evaluating the chinese education values of large language models, 2025

    Peiyi Zhang, Yazhou Zhang, Bo Wang, Lu Rong, Prayag Tiwari, and Jing Qin. Edu-values: Towards evaluating the chinese education values of large language models, 2025

    2025

  67. [75]

    Evaluation and mitigation of cognitive biases in medical language models.npj Digital Medicine, 7(1):295, 2024

    Samuel Schmidgall, Carl Harris, Ime Essien, Daniel Olshvang, Tawsifur Rahman, Ji Woong Kim, Rojin Ziaei, Jason Eshraghian, Peter Abadir, and Rama Chellappa. Evaluation and mitigation of cognitive biases in medical language models.npj Digital Medicine, 7(1):295, 2024

    2024

  68. [76]

    Socioeconomic status and mental health — Wikipedia, the free encyclopedia,

    Wikipedia contributors. Socioeconomic status and mental health — Wikipedia, the free encyclopedia,

  69. [77]

    Glover, Diana M

    John D. Glover, Diana M. Hetzel, and Sarah K. Tennant. The socioeconomic gradient and chronic illness and associated risk factors in australia.Australia and New Zealand Health Policy, 1(1):8, 2004. PMID: 15679942, PMCID: PMC546403

    2004

  70. [78]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

    2011

  71. [79]

    Lg-cav: Train any concept activation vector with language guidance

    Qihan Huang, Jie Song, Mengqi Xue, Haofei Zhang, Bingde Hu, Huiqiong Wang, Hao Jiang, Xingen Wang, and Mingli Song. Lg-cav: Train any concept activation vector with language guidance. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors...

    2024

  72. [80]

    describe

    Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. Saelens. https://github.com/ jbloomAus/SAELens, 2024. 15 A Limitations BIASLENSuses a single prompt for CA V-based steering, though in practice multiple prompts may satisfy the steering criteria listed in Appendix B.4...

    2024

  73. [84]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  74. [2021]

    Association for Computational Linguistics

  75. [2024]

    [Online; accessed 8-May-2025]

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.