REVIEW 3 major objections 4 minor 17 references
A debiasing method for LLMs that amplifies the neurons encoding knowledge of bias, claiming consistent bias reduction with little loss of general ability.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 06:42 UTC pith:GHL4N4J2
load-bearing objection A practical, cheap debiasing trick with broad empirical support, but the 'know-bias neuron' story is under-tested and the headline benchmark is partly tuned. the 3 major comments →
Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central discovery claim is that social bias in LLMs can be mitigated by strengthening, not suppressing, the internal representations that encode knowledge of bias. The authors designate these as know-bias neurons: intermediate feed-forward neurons whose activations are most responsible, under integrated-gradient attribution, for a model answering simple bias-knowledge questions as a normatively informed model would—rejecting causal links between demographics and abilities, recognizing biased beliefs as biased, and judging them morally wrong. At inference, all selected neurons are scaled by a factor λ, with no parameter updates. The paper reports that this intervention reduces ste
What carries the argument
Know-bias neurons: a small set of intermediate feed-forward neurons selected by integrated-gradient attribution on simple yes/no bias-knowledge questions (causal rejection, bias recognition, normative judgment) across gender, race, and religion, aggregated by union across dimensions. The mechanism is a single multiplicative activation boost h ← λh applied to exactly those neurons at decoding time, with no retraining or parameter editing; the claimed work is that this amplification activates the model's latent normative knowledge and thereby changes choice probabilities toward neutral alternatives.
Load-bearing premise
The load-bearing premise is that neurons singled out by attribution to a few dozen probe questions are the same neurons whose amplification steers ordinary text generation toward fairer outputs—and not merely that the model has been nudged toward saying 'No'.
What would settle it
Compute the attribution with every expected answer flipped (e.g., target 'Yes' for causal-rejection questions); if the flipped neuron set debiases equally well on the benchmarks, the method is selecting answer-phrasing neurons rather than bias-knowledge neurons; if debiasing collapses, the specific normative answers are doing the causal work.
If this is right
- If the transfer results hold, one probe set built from a few abstract bias concepts can debias many stereotypes: race/problem-solving probes shifted probabilities on crime, wealth, and appearance stereotypes and on multiple demographic identities.
- Because the intervention is applied only at inference, the same model can be run debiased or not, and the neuron set can be recomputed cheaply when the deployment context changes.
- The saturation at roughly 45 questions implies the method's data cost is the cost of writing three question templates, not collecting thousands of stereotype-labeled examples, lowering the barrier to bias mitigation for smaller teams.
- Preserved general reasoning (reported on standard benchmarks such as OpenBookQA, COPA, and ARC) suggests a more favorable fairness-utility trade-off than suppression-based editing or fine-tuning, making the approach viable for production deployments.
- The paper itself flags that the same 'enhance knowledge, don't suppress behavior' recipe could be applied to other alignment goals such as safety and honesty.
Where Pith is reading between the lines
- My inference: the method's real test is whether the selected neurons are bias-knowledge-specific. A cheap way to check is to flip the expected answers and see whether the resulting neuron set still debiases; if it does, the attribution may be tracking a generic 'No' response rather than normative knowledge.
- My inference: because the paper's impact statement concedes that the yes/no answers encode the prompt designers' normative judgments and the model's pretraining norms, the method steers models toward those embedded norms rather than toward an objective standard of fairness; cross-cultural recalibration of the question set would be needed before deployment across societies.
- My inference: the three demographic dimensions tested—gender, race, and religion—are the standard set in the benchmarks, but the same three question templates could be run for disability, age, nationality, or socioeconomic status; the open question is whether the existing unified neuron set already covers those axes or fresh probes are required.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes KnowBias, an inference-time debiasing framework that identifies 'know-bias neurons' using attribution scores computed on a small set of 45 hand-written yes/no bias-knowledge questions (Eq. 2), and then multiplicatively scales those neurons by a factor λ during generation. The authors report consistent state-of-the-art debiasing across BBQ, CrowS-Pairs, and StereoSet (intra-sentence and inter-sentence) on three instruction-tuned LLMs, while largely preserving general reasoning ability on OBQA, COPA, and ARC. They also present evidence for cross-dimension transfer, data efficiency, and ablations including a random-neuron control. The central claim is that amplifying neurons that encode normative bias knowledge—rather than suppressing bias-behavior neurons—causally steers the model toward fairer outputs without retraining.
Significance. If the causal claim holds, KnowBias would be a valuable contribution: it offers a lightweight, retraining-free alternative to suppressive debiasing, with potential interpretability (a small set of neurons) and cross-demographic transfer. The paper provides broad empirical support: three backbones, five bias benchmarks, four utility benchmarks, ablations that include a random-neuron baseline (Table 9), and public code/data. The main weaknesses are that the headline results are partly fitted via hyperparameter selection on StereoSet-Inter, and that the attribution target—a single expected Yes/No token—leaves the 'bias-knowledge' interpretation of the selected neurons underdetermined. Both issues are addressable with additional validation and control experiments, but they currently block a clean acceptance.
major comments (3)
- [Appendix D.7, Figure 4; Table 2] The hyperparameters q, τ, β, and λ are selected by maximizing StereoSet-Inter ICAT (Figure 4a–d) on Llama-3.2-3B-Instruct, and SS-inter is one of the headline metrics in Table 2. This is a selection-on-the-test-metric problem: the reported SS-Inter improvements are in-sample, and the claim of 'consistent state-of-the-art' across benchmarks is weakened because the method was tuned on one of those benchmarks. Please provide a proper validation split, pre-register hyperparameters, or report results under a fixed setting; also state explicitly how the per-backbone settings in Appendix D.3 were chosen.
- [Eq. (2), Section 2.3; Table 9] The attribution score measures contribution to the probability of a single expected token a* ∈ {Yes, No}. With τ=10% and β=10%, the selected set may be dominated by neurons that generically boost the expected answer token or safe/neutral continuations, rather than neurons encoding bias knowledge. The random-neuron baseline (Table 9) rules out random scaling but not attribution-selection on arbitrary probes. A control using factual yes/no questions (e.g., 'Is the sky blue?') or non-bias normative questions, plus a check of whether the same neuron set changes answers to arbitrary yes/no questions, is needed to support the causal 'know-bias neuron' interpretation. Cross-dimension transfer (Table 5) is also consistent with generic answer-compliance neurons.
- [Appendix D.3] The main text reports different hyperparameters for Qwen-3-4B (λ=3.5) and Llama-3.1-8B (β=5%, λ=2), but the hyperparameter study in Appendix D.7 is only shown for Llama-3.2-3B-Instruct. It is unclear whether these settings were also tuned on SS-Inter, and whether the claimed generalization across backbones holds under a uniform, pre-specified protocol. Report the selection procedure for each model, or provide sensitivity results for the other two backbones.
minor comments (4)
- [Section 3.4] The text refers to 'CRISP' in one place and 'CRISPR' elsewhere; the baseline is CRISPR (Yang et al., 2024b). Please make the naming consistent.
- [Appendix D.3] The model name is written 'Llama3.2-3B' in one paragraph and 'Llama-3.2-3B' elsewhere; unify the notation.
- [Section 3.6 / Appendix D.7] The hyperparameter section says results are in Appendix D.7, but Figure 4 only covers Llama-3.2-3B. The statement in D.7.5 that 'complete hyperparameter settings for all three models are in Appendix D.3' is misleading because D.3 lists only the final chosen values, not sensitivity studies. Clarify the coverage.
- [Table 4] The qualitative examples are useful, but the probabilities are not accompanied by an aggregate statistic across the SS-inter-race subset. A quantified summary (e.g., average probability shift toward neutral) would strengthen the claim of generalizable debiasing.
Circularity Check
SS-Inter results are partly fitted: q, tau, beta, and lambda are selected on SS-Inter itself, then the same SS-Inter scores are reported as evidence; other benchmarks remain external.
specific steps
-
fitted input called prediction
[Section 3.6 / Appendix D.7 (Figure 4); results reported in Table 2]
"Hyperparameter study for KnowBias on Llama-3.2-3B-Instruct. ICAT score (↑) for Stereoset-Inter. ... As shown in Figure 4a, debiasing performance improves rapidly as q increases ... saturates once q reaches approximately 45 questions ... Thus, we pick q= 45 as an optimal value. ... The results in Figure 4b show that τ= 10% consistently achieves the strongest debiasing performance ... The results (Figure 4d) show that λ= 2 consistently yields the best debiasing performance..."
The hyperparameters q, tau, beta, and lambda are chosen by maximizing ICAT on StereoSet-Inter (Figure 4), and the same SS-Inter ICAT values appear in Table 2 as part of KnowBias's claimed state-of-the-art debiasing performance. The reported SS-Inter improvement is therefore the selection objective, not an independent prediction. BBQ, CrowS-Pairs, and general-capability results were not used for tuning and remain out-of-sample, so the circularity is partial rather than total.
full rationale
The conceptual derivation is not circular: bias-knowledge questions are external to the bias benchmarks, attribution scores are computed from those questions via integrated gradients (Eq. 2), and inference-time scaling is defined independently of the evaluation metrics. Self-citations (Pan et al. 2025a,b) are used only as background on bias prevalence and dimension choice, not as load-bearing evidence. The one genuine circular component is the hyperparameter study: q=45, tau=10%, beta=10%, and lambda=2 are tuned on SS-Inter ICAT in Figure 4, and the resulting SS-Inter numbers are then presented in Table 2 as evidence of SOTA performance. This makes the SS-Inter portion of the headline claim fitted rather than predicted, while the cross-benchmark claims retain independent content.
Axiom & Free-Parameter Ledger
free parameters (4)
- enhancement scale lambda =
2 (Llama-3.2/3.1-8B), 3.5 (Qwen-3-4B)
- attribution threshold tau =
10%
- cross-question frequency threshold beta =
10% (Llama-3.2/Qwen), 5% (Llama-3.1-8B)
- number of bias-knowledge questions q =
45 (5 concepts x 3 question types x 3 demographics)
axioms (5)
- ad hoc to paper LLMs encode normative bias knowledge in a shared set of FFN neurons such that amplifying them reduces biased generation.
- domain assumption Integrated-gradient attribution on the expected yes/no answer identifies the neurons causally responsible for bias knowledge (Eq. 2, Section 2.3).
- domain assumption Multiplicatively scaling a small set of neurons by lambda > 1 preserves general capabilities.
- ad hoc to paper Three hand-written question types and five abstract bias concepts are a sufficient probe of generalizable bias knowledge.
- domain assumption Transformer MLP neurons behave as key-value memories (Geva et al. 2021; Dai et al. 2022).
invented entities (1)
-
know-bias neurons
no independent evidence
Cite this review
Pith. "Pith review of Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement." pith.science (2026). https://pith.science/paper/GHL4N4J2
@misc{pith2026260121864,
author = {Pith},
title = {Pith review of: Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHL4N4J2}},
note = {Machine review of arXiv:2601.21864}
}
read the original abstract
Large language models (LLMs) exhibit social biases that reinforce harmful stereotypes, limiting their safe deployment. Most existing debiasing methods adopt a suppressive paradigm by modifying parameters, prompts, or neurons associated with biased behavior; however, such approaches are often brittle, weakly generalizable, data-inefficient, and prone to degrading general capability. We propose \textbf{KnowBias}, a lightweight and conceptually distinct framework that mitigates bias by strengthening, rather than suppressing, neurons encoding bias-knowledge. KnowBias identifies neurons encoding bias knowledge using a small set of bias-knowledge questions via attribution-based analysis, and selectively enhances them at inference time. This design enables strong debiasing while preserving general capabilities, generalizes across bias types and demographics, and is highly data efficient, requiring only a handful of simple yes/no questions and no retraining. Experiments across multiple benchmarks and LLMs demonstrate consistent state-of-the-art debiasing performance with minimal utility degradation. Data and code are available at https://github.com/JP-25/KnowBias.
Figures
Reference graph
Works this paper leans on
-
[1]
Know-bias neuron identification. Initialize empty neuron setsN gender,N race,N religion ford∈ {gender,race,religion}do foreach questionbq∈ Qd with the corresponding target tokenado Compute attribution scores{α (l) i } ←ATTR(θ, bq, a)for all neurons, wherebqis the input prompt here. end for Select neurons Nd based on attribution scores, attribution thresho...
-
[2]
Nknow-bias ← Ngender ∪ Nrace ∪ Nreligion
Neuron set aggregation. Nknow-bias ← Ngender ∪ Nrace ∪ Nreligion
-
[3]
Do you think that gender/race/religion influences problem-solving skills?
Inference-time enhancement. forlayerland neuroni∈ N know-bias do h(l) i ←λ·h (l) i end for for every intermediate neuron across all prompts expressing the same underlying behavior. Neurons whose scores exceed the attribution threshold τ% of the maximum score across the whole model and that are consistently shared for at least β% of the prompts are selecte...
arXiv 2022
-
[6]
Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl
-
[13]
Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long
-
[43]
URL https://aclanthology.org/2023. findings-eacl.43/. Yang, N., Kang, T., Choi, S. J., Lee, H., and Jung, K. Miti- gating biases for instruction-following language models via bias neurons elimination. In Ku, L.-W., Martins, A., and Srikumar, V . (eds.),Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...
-
[74]
naacl-short.74/
URL https://aclanthology.org/2025. naacl-short.74/. Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. InProceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484–5495, 2021. Gonc ¸alves, G. and Strubell, E. Understanding the effect of model compression on socia...
2025
-
[154]
URL https://aclanthology.org/2020. emnlp-main.154/. Navigli, R., Conia, S., and Ross, B. Biases in large language models: Origins, inventory, and discussion.ACM Journal of Data and Information Quality, 15(2):10:1–10:21, 2023. URLhttps://doi.org/10.1145/3597307. Niu, J., Liu, A., Zhu, Z., and Penn, G. What does the knowledge neuron thesis have to do with k...
Pith/arXiv arXiv 2020
-
[161]
URL https://aclanthology.org/2023. emnlp-main.161/. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. Hofmann, V ., Kalluri, P. R., Jurafsky, D., and King, S. Ai generates covertly racist decisions about p...
Pith/arXiv arXiv 2023
-
[165]
URL https://aclanthology.org/2022. findings-acl.165/. Qin, Z., Ding, Y ., Liu, D., Liu, Q., Cai, J., Chen, X., Tu, Z., Chu, D., Gao, C., and Sui, D. Lftf: Locating first and then fine-tuning for mitigating gender bias in large language models.arXiv preprint arXiv:2505.15475, 2025a. Qin, Z., Lyu, K., Yu, Q., Sun, Y ., and Fan, Z. The achilles’ heel of llms...
Pith/arXiv arXiv 2022
-
[191]
URL https://aclanthology.org/2024. emnlp-main.191/. Yu, Z. and Ananiadou, S. Understanding and mitigating gender bias in llms via interpretable neuron editing.arXiv preprint arXiv:2501.14457, 2025. Zhang, Z., Zhao, J., Zhang, Q., Gui, T., and Huang, X. Un- veiling linguistic regions in large language models. In Ku, L.-W., Martins, A., and Srikumar, V . (e...
Pith/arXiv arXiv 2024
-
[485]
URL https://aclanthology.org/2020. acl-main.485/. Cheng, X., Chen, R., Zan, H., Jia, Y ., and Peng, M. BiasFil- ter: An inference-time debiasing framework for large lan- guage models. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V . (eds.),Findings of the Asso- ciation for Computational Linguistics: EMNLP 2025, pp. 15187–15205, Suzhou, ...
-
[490]
URL https://aclanthology.org/2024. acl-long.490/. 11 KnowBias: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement Yu, Z. and Ananiadou, S. Neuron-level knowledge at- tribution in large language models. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processi...
-
[589]
URL https://aclanthology.org/2025. findings-acl.589/. Ma, X., Liu, Q., Jiang, D., Zhang, G., Ma, Z., and Chen, W. General-reasoner: Advancing llm reasoning across all domains, 2025. URL https://arxiv.org/abs/ 2505.14652. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answe...
Pith/arXiv arXiv 2025
-
[717]
knowledge neurons
URL https://aclanthology.org/2025. acl-long.717/. 12 KnowBias: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement A. Related Work A.1. Social Bias Mitigation in LLMs Social bias in LLMs has become a critical concern (Hofmann et al., 2024; Navigli et al., 2023; Cui et al., 2024). Existing debiasing methods can be broadly categorized into four ...
2025
-
[821]
URL https://aclanthology.org/2025. findings-emnlp.821/. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. Crawford, K. The trouble with bias. Talk at NeurIPS, December 2017. URL http://youtube.com/ watch?v=fMy...
Pith/arXiv arXiv 2025
-
[2025]
Association for Computational Linguistics. ISBN 979-8-89176-190-2. doi: 10.18653/v1/2025.naacl-short
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.