Pith. sign in

REVIEW 5 major objections 5 minor 61 references

Truth Neurons

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language models contain 'truth neurons' whose suppression degrades truthful answer selection across benchmarks.

desk verdict A new integrated-gradients pipeline for locating truth-selective neurons, with a plausible but unproven central claim; the suppression evidence is weakened by threshold tuning and missing task-criticality controls. read the letter →

arxiv 2505.12182 v3 pith:JTN3EYTT submitted 2025-05-18 cs.CL

Haohang Li , Yupeng Cao , Yangyang Yu , Jordan W. Suchow , Zining Zhu This is my paper · ORCID
classification cs.CL
keywords truthneuronsneuronattributionintegratedgradientstruthfulnessmechanisticinterpretabilitylanguagemodelsTruthfulQAsuppression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Transformer-based language models contain sparse, identifiable 'truth neurons' whose activations encode truthfulness in a subject-agnostic way. It introduces an integrated-gradients attribution procedure that scores every feed-forward and attention neuron by how much it raises the probability of the correct answer relative to an incorrect one, then filters to a small, statistically consistent set. Zeroing those neurons—but not an equal number of random neurons—drops TruthfulQA accuracy in all six models tested (by roughly 1 to 19 percentage points), and the same neurons reduce accuracy on TriviaQA and MMLU in eleven of twelve model–benchmark pairs. The identified neurons concentrate in the middle layers, consistent with earlier evidence that true and false statements are linearly separable in hidden states. A reader should care because the result points to a concrete, intervenable substrate for one of LLMs' most safety-relevant behaviors: whether the model answers truthfully.

What carries the argument

The central object is the truth neuron, defined through the attribution difference $D(n_{i,l}) = \operatorname{Attr}_t(n_{i,l}) - \operatorname{Attr}_f(n_{i,l})$ for neuron $i$ in layer $l$, where the two terms are integrated-gradient scores for the correct and incorrect answer probabilities. Integrated gradients integrate the gradient of each output probability along a straight-line path from a baseline activation to the observed activation, so the difference measures how strongly a neuron shifts probability toward truth and away from falsehood. Candidate neurons must pass a maximum-activation threshold and a consistency threshold (salient for at least 40% of examples), and a split-half design—selection on one half, significance testing on the other—guards against double-dipping. Suppressing exactly these neurons is the causal probe: the same intervention on randomly chosen neurons does not reproduce the accuracy drop.

What would settle it

Re-run the entire pipeline on the same six models with a pre-registered threshold rule—for example, one fixed threshold for all models, or a threshold chosen without looking at the suppression effect—and check whether zeroing the selected neurons still gives a statistically significant accuracy drop relative to random-neuron suppression; if it does not, the central causal claim is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that truthfulness is not a diffuse property of the whole network: it is carried by a small number of neurons whose activations causally push the model toward correct answers. For each of six instruction-tuned models, the method isolates 35 to 655 candidate neurons by taking the difference between integrated-gradient attributions for the correct and incorrect answers in a binary-choice setup, correcting for multiple comparisons, and requiring each neuron to be consistently salient across at least 40% of examples. When these neurons' activations are set to zero, TruthfulQA accuracy drops for every model, with reductions from about 1.4 to 18.8 percentage points, while the average probability assigned to the correct answer drops by 22.1%. The neurons selected from TruthfulQA also reduce accuracy on MMLU and TriviaQA in eleven of twelve model–benchmark pairs, which the paper takes as evidence that the representation is dataset-general rather than benchmark-specific.

Load-bearing premise

The result rests on a per-model threshold that is manually adjusted until suppressing the selected neurons produces a noticeable accuracy change, so the reported significance is partly a product of that fitting rather than a fully a priori criterion.

Editorial extensions

If this is right

  • Zeroing the identified neurons is a minimal, targeted intervention that moves six different models toward untruthful answers, so truthfulness can be studied and manipulated without retraining.
  • Because neurons selected on TruthfulQA also degrade MMLU and TriviaQA, the mechanism appears to be a shared internal notion of factual correctness rather than a dataset artifact.
  • The layer distribution gives a concrete search target: truthfulness machinery sits mainly in middle layers, with secondary clusters deeper in the network.
  • Same-family models show similar suppression magnitudes, suggesting that training-data distribution helps determine where truth neurons form.
  • Truth-neuron counts and locations could serve as a mechanistic diagnostic for auditing a model's truthfulness before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the neurons are truly subject-agnostic truth encoders, suppressing them should also reduce truthfulness in free-form generation and non-English benchmarks; the paper does not test either setting.
  • Inference: because the per-model threshold is tuned until the effect appears, an alternative reading is that the selected set partly reflects the tuning procedure; a pre-registered threshold rule would discriminate the two interpretations.
  • Inference: the same identification pipeline could be run in reverse—amplifying rather than suppressing truth neurons—to test whether truthfulness can be boosted without the side effects of prompt-level interventions.
  • Inference: the middle-layer clustering connects to linear-probe results, which suggests comparing the truth-neuron set to the top directions of a linear truth probe at the same layer; overlapping supports the causal reading, disjoint sets would complicate it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper claims that language models contain 'truth neurons' that encode truthfulness in a subject-agnostic manner. The authors propose an integrated-gradient attribution method on a binary-choice TruthfulQA task, where the 'truthful' class is the correct answer and the 'untruthful' class is the incorrect answer. They select neurons whose average attribution difference is significantly positive and consistently salient across examples, then suppress these neurons and measure accuracy drops on TruthfulQA, TriviaQA, and MMLU across six models. They report statistically significant drops on TruthfulQA, apparent generalization to the other benchmarks, and a layer distribution concentrated in middle and later layers. The paper concludes that these results establish the existence of neuron-level truthfulness mechanisms shared across models.

Significance. If the claim were established, a neuron-level account of truthfulness in language models would be a substantial contribution to mechanistic interpretability and could inform safety-relevant interventions. The paper has some strengths: it evaluates six models of varying scales, uses an established attribution method (integrated gradients), includes a random-neuron control in the primary suppression experiment, and attempts to address double-dipping by splitting the data for the statistical test. However, the current evidence does not rule out the alternative hypothesis that the selected neurons are merely task-critical for binary question answering rather than specifically truth-encoding. Because the central interpretation rests on controls and evaluation procedures that are not adequate for the claim, the significance of the finding as presented is limited.

major comments (5)
  1. [Section 3.1 / Appendix A.3] The adaptive threshold t% is manually tuned until a noticeable performance difference is achieved: Appendix A.3 states that the authors 'iteratively adjusted it until we achieved a noticeable performance difference while preserving the model’s ability to follow instructions.' This makes the statistically significant suppression effects in Table 1 partly a product of selecting the threshold on the outcome. The thresholds should be fixed a priori, or tuned on a validation split that is not used for the reported evaluation.
  2. [Section 2.2 / Section 2.3] The selection score D(n_i,l) = Attr_t - Attr_f measures a neuron's contribution to choosing the correct answer over the incorrect answer in a binary-choice task, and the manipulation check retains only questions the model can answer correctly. Consequently, any neuron critical for general binary question-answering (e.g., instruction following, option selection, reasoning) will receive a large positive score. The random-neuron control in Table 1 suppresses uniformly sampled neurons, which are almost always low-salience; it does not control for the importance of the selected neurons. A control that suppresses an equal number of matched high-salience neurons (e.g., selected by absolute attribution or by attribution on class labels) is needed to support the claim that the effect is specific to truthfulness rather than to task competence.
  3. [Section 3.4 / Table 2] The generalization experiments in Table 2 omit even a random-neuron control, so the observed drops on TriviaQA and MMLU are consistent with non-specific suppression damage. Furthermore, Qwen2.5-3B-Instruct's MMLU accuracy increases from 62.10 to 62.70 after suppression, which the text acknowledges as an exception; this is incompatible with the strong claim that these neurons encode truthfulness in a subject-agnostic manner and further suggests the effects are not uniformly truth-specific.
  4. [Section 3.3 / Figure 4] The claim that truth neurons encode category-agnostic truthfulness is based on probability decreases across TruthfulQA categories, but this analysis uses the same TruthfulQA data from which the neurons were selected. It therefore does not provide independent evidence of subject-agnostic encoding; a held-out dataset with a different topic distribution would be needed.
  5. [Section 2.3] The paper states that the dataset is split into two halves for neuron selection and for the t-test, but it does not clarify whether the suppression evaluation in Table 1 also uses a held-out split or the full dataset. If the same full TruthfulQA set is used for both selecting neurons and measuring the suppression effect, the causal test overlaps with the selection data, undermining the independence of the evidence. Please clarify the exact split used for the suppression experiment.
minor comments (5)
  1. [Equation (3)] There is a typographical issue in the text near Equation (3): 'withnbaseline' should be 'with n_baseline' to be readable.
  2. [Figure 3 / Equation (11)] The caption says 'Values are reported as percentages (%)' but Eq. (11) is a dimensionless ratio; please clarify whether the values are multiplied by 100 and define the quantity precisely.
  3. [Table 2] Unlike Table 1, Table 2 reports no standard deviations or repetitions; adding uncertainty estimates would help assess the robustness of the generalization results.
  4. [References [11] and [45]] References [11] and [45] are informal web sources (Alignment Forum and a personal blog); the authors should cite peer-reviewed versions or justify the reliance on these sources.
  5. [Section 2.2] The term 'truth neuron' is introduced as a construct but is defined only operationally through the selection procedure; a more explicit definition that distinguishes it from neurons that are merely task-critical would strengthen the paper.

Circularity Check

2 steps flagged · score 6.0 of 10

Truth neurons are selected as correct-answer-supporting neurons via an attribution difference, and the threshold controlling the suppression effect is tuned until a noticeable drop appears; the central claim is therefore partly circular, though OOD transfer adds some independent content.

  1. self definitional [Section 2.2, Eq. (8) and Notation; Section 3.3 (suppression results)]
    "For each question q, the dataset provides one correct answer t and one incorrect answer f... For a given neuron ni,l... applying integrated gradients to the input with respect to the correct and incorrect responses yields Attrt(ni,l| f(T|t)) and Attrf(ni,l| f(T|f))... D(ni,l) = Attrt(ni,l| f(T|t))− Attrf(ni,l| f(T|f))"

    The paper defines 'truthful' as the correct answer and 'untruthful' as the incorrect answer in a binary-choice setup. The selection score D is by construction the neuron's attribution toward the correct response minus its attribution toward the incorrect response. Neurons with large positive D are then labeled 'truth neurons,' and suppressing them is reported to reduce accuracy on the same binary-choice task. This outcome is largely a restatement of the selection criterion: neurons chosen as most supportive of correct-over-incorrect choice should, when ablated, reduce correct choice. The random-neuron control matches only the number of suppressed neurons, not their task-salience, so it does not break the construction.

  2. fitted input called prediction [Appendix A.3, Table 3]
    "Since attribution scales vary across models, the adaptive threshold (t%) need be manually configured. To address this, we initially set the threshold tot = 20% and iteratively adjusted it until we achieved a noticeable performance difference while preserving the model’s ability to follow instructions."

    The hyperparameter that determines how many neurons are selected, and therefore the magnitude of any suppression effect, is manually tuned until a noticeable performance difference is observed. Reporting the resulting accuracy drop as evidence for the existence of truth neurons is partly a product of this fitting: there is no fixed a-priori threshold, and the significance of the effect is not an independent prediction. The threshold is effectively fit to the outcome it is later used to demonstrate.

full rationale

The paper's central derivation has two partially circular components. First, the definition of a truth neuron is itself based on the correct-versus-incorrect answer contrast in TruthfulQA, and the main suppression result on TruthfulQA is therefore closely tied to that definition. This is not a pure tautology because integrated-gradient attribution does not guarantee a causal suppression effect, but it makes the headline result substantially by construction. Second, the adaptive threshold t% is explicitly tuned until a noticeable performance difference is achieved, so the reported effect size is partly a fitting artifact. These two issues justify a score of 6. The OOD generalization to TriviaQA and MMLU provides some independent evidence, and there is no load-bearing self-citation chain or imported uniqueness theorem; those aspects are not circular. The paper does attempt to avoid double-dipping by splitting selection and statistical testing, but the threshold tuning and the definitional link between 'truth' and 'correct answer' remain.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the validity of integrated gradient attribution as a causal measure, the binary-choice benchmark as a truthfulness proxy, and a filtering procedure that includes a manually tuned threshold. The threshold is the most serious free parameter because it is adjusted to produce a significant effect.

free parameters (3)
  • Adaptive threshold t% = 1 (Qwen2.5-3B), 20 (Llama-3.2-3B), 10 (OLMo-7B), 25 (Llama-3.1-8B), 17 (Mistral-Nemo), 20 (OLMo-13B)
    Tuned per model to achieve a noticeable suppression effect, which risks selecting thresholds that produce a significant result.
  • Share threshold p = 40%
    Set to require neurons to be salient across 40% of examples; chosen a priori but no sensitivity analysis.
  • Interpolation steps m = 20
    Riemann sum steps for integrated gradients; reasonable but arbitrary.
assumptions (3)
  • domain assumption Integrated gradient attribution to a neuron's activation correctly measures that neuron's contribution to the model's output probability for a token.
    The method relies on this to identify candidate neurons, but integrated gradients is an approximation and not a causal measure.
  • domain assumption The binary-choice TruthfulQA evaluation (Evans et al.) accurately reflects truthfulness.
    The paper adopts the updated binary format but this is still a proxy for truthfulness.
  • ad hoc to paper Only examples where the model answers correctly are informative for truth neuron identification.
    The manipulation check filters examples, which could exclude cases where truthfulness mechanisms are most relevant.
invented entities (1)
  • Truth neuron
    purpose: A neuron that encodes truthfulness in a subject-agnostic manner
    The existence is inferred from attribution and suppression on the same task family; no independent evidence distinguishes truth specificity from general task importance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Truth Neurons." pith.science (2026). https://pith.science/paper/JTN3EYTT

@misc{pith2026250512182,
  author       = {Pith},
  title        = {Pith review of: Truth Neurons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTN3EYTT}},
  note         = {Machine review of arXiv:2505.12182}
}
read the original abstract

Despite their remarkable success and deployment across diverse workflows, language models sometimes produce untruthful responses. Our limited understanding of how truthfulness is mechanistically encoded within these models jeopardizes their reliability and safety. In this paper, we propose a method for identifying representations of truthfulness at the neuron level. We show that language models contain truth neurons, which encode truthfulness in a subject-agnostic manner. Experiments conducted across models of varying scales validate the existence of truth neurons, confirming that the encoding of truthfulness at the neuron level is a property shared by many language models. The distribution patterns of truth neurons over layers align with prior findings on the geometry of truthfulness. Selectively suppressing the activations of truth neurons found through the TruthfulQA dataset degrades performance both on TruthfulQA and on other benchmarks, showing that the truthfulness mechanisms are not tied to a specific dataset. Our results offer novel insights into the mechanisms underlying truthfulness in language models and highlight potential directions toward improving their trustworthiness and reliability.

Figures

Figures reproduced from arXiv: 2505.12182 by the authors.

Figure 1
Figure 1. Overview of our method that detects the truth neurons. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of attribution score computation. The left side shows example attribution scores [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average change in the probability of cor [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Proportion of questions within each category for which the probability of selecting the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Distribution of identified truth neurons across layers for different language models. Each [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Prompt template used to generate adversarial answers for TriviaQA questions. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt template used for TriviaQA evaluation. The correct answer and incorrect answer [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt template used for MMLU evaluation. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt template used for TruthfulQA evaluation. The correct answer and incorrect answer [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 36 canonical work pages

  1. [1]

    The internal state of an LLM knows when it’s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 967–976, 2023

    work page 2023

  2. [2]

    Enhanced language model truthfulness with learnable intervention and uncertainty expression

    Farima Fatahi Bayat, Xin Liu, H Jagadish, and Lu Wang. Enhanced language model truthfulness with learnable intervention and uncertainty expression. In Findings of the Association for Computational Linguistics ACL 2024, pages 12388–12400, 2024. 9

    work page 2024

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  4. [4]

    Truth is universal: Robust detection of lies in llms

    Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in llms. Advances in Neural Information Processing Systems , 37:138393–138431, 2024

    work page 2024

  5. [5]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2024

    work page 2024

  6. [6]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology , 15(3):1–45, 2024

    2024

  7. [7]

    Inside: Llms’ internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Llms’ internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations , 2024

    work page 2024

  8. [8]

    Dola: Decoding by contrasting layers improves factuality in large language models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

Show all 61 references
  1. [9]

    Knowledge neurons in pretrained transformers

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021

    2021 arXiv

  2. [10]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  3. [11]

    New, improved multiple-choice truthfulqa

    Owain Evans, James Chua, and Steph Lin. New, improved multiple-choice truthfulqa. https://www.alignmentforum.org/posts/Bunfwz6JsNd44kgLT/ new-improved-multiple-choice-truthfulqa , 2025. Accessed: 2025-05-08

    2025

  4. [12]

    Do i know this entity? knowledge awareness and hallucinations in language models

    Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257, 2024

    2024 arXiv

  5. [13]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021

    2021

  6. [14]

    The llama 3 herd of models, 2024

    Aaron Grattafiori and etal. The llama 3 herd of models, 2024. URL https://arxiv.org/ abs/2407.21783

    2024 arXiv

  7. [15]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https: //arxiv.org/abs/2009.03300

    2021 arXiv

  8. [16]

    Open-finllms: Open multimodal large language models for financial applications

    Jimin Huang, Mengxi Xiao, Dong Li, Zihao Jiang, Yuzhe Yang, Yifei Zhang, Lingfei Qian, Yan Wang, Xueqing Peng, Yang Ren, et al. Open-finllms: Open multimodal large language models for financial applications. arXiv preprint arXiv:2408.11878, 2024

    2024 arXiv

  9. [17]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informat...

    2025

  10. [18]

    Position: Trustllm: Trustworthiness in large language models

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Position: Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning , pages 20166–20270. PMLR, 2024. 10

    2024

  11. [19]

    The evolution of numerical cognition: From number neurons to linguistic quantifiers

    Edward M Hubbard, Ilka Diester, Jessica F Cantlon, Daniel Ansari, Filip Van Opstal, and Vanessa Troiani. The evolution of numerical cognition: From number neurons to linguistic quantifiers. Journal of Neuroscience, 28(46):11819–11824, 2008

    2008

  12. [20]

    Single-neuronal predictions of others’ beliefs in humans.Nature, 591(7851):610–614, 2021

    Mohsen Jamali, Benjamin L Grannan, Evelina Fedorenko, Rebecca Saxe, Raymundo Báez- Mendoza, and Ziv M Williams. Single-neuronal predictions of others’ beliefs in humans.Nature, 591(7851):610–614, 2021

    2021

  13. [21]

    Cognitive neuroscience of honesty and deception: A signaling framework

    Adrianna C Jenkins, Lusha Zhu, and Ming Hsu. Cognitive neuroscience of honesty and deception: A signaling framework. Current Opinion in Behavioral Sciences , 11:130–137, 2016

    2016

  14. [22]

    A survey on large language models for code generation

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024

    2024 arXiv

  15. [23]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  16. [24]

    Brain network transformer

    Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems , 35:25586–25599, 2022

    2022

  17. [25]

    SwiFT: Swin 4D fMRI Transformer

    Peter Kim, Junbeom Kwon, Sunghwan Joo, Sangyoon Bae, Donggyu Lee, Yoonho Jung, Shinjae Yoo, Jiook Cha, and Taesup Moon. SwiFT: Swin 4D fMRI Transformer. Advances in Neural Information Processing Systems, 36:42015–42037, 2023

    2023

  18. [26]

    Circular analysis in systems neuroscience: the dangers of double dipping

    Nikolaus Kriegeskorte, W Kyle Simmons, Patrick SF Bellgowan, and Chris I Baker. Circular analysis in systems neuroscience: the dangers of double dipping. Nature neuroscience, 12(5): 535–540, 2009

    2009

  19. [27]

    Shared functional specialization in transformer-based language models and the human brain

    Sreejan Kumar, Theodore R Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Ken- neth A Norman, Thomas L Griffiths, Robert D Hawkins, and Samuel A Nastase. Shared functional specialization in transformer-based language models and the human brain. Nature communications, ...

    2024

  20. [28]

    Pre-trained language models for text generation: A survey

    Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1–39, 2024

    2024

  21. [29]

    Inference- time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530, 2023

    2023

  22. [30]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  23. [31]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9004–9017, 2023

    2023

  24. [32]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  25. [33]

    What does the knowledge neuron thesis have to do with knowledge? arXiv preprint arXiv:2405.02421, 2024

    Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? arXiv preprint arXiv:2405.02421, 2024

    2024 arXiv

  26. [34]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2025 arXiv

  27. [35]

    LLMs know more than they show: On the intrinsic representation of LLM hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. arXiv preprint arXiv:2410.02707, 2024

    arXiv

  28. [36]

    Ai deception: A survey of examples, risks, and potential solutions

    Peter S Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Patterns, 5(5), 2024

    2024

  29. [37]

    Human single-neuron responses at the threshold of conscious recognition

    R Quian Quiroga, Roy Mukamel, Eve A Isham, Rafael Malach, and Itzhak Fried. Human single-neuron responses at the threshold of conscious recognition. Proceedings of the National Academy of Sciences, 105(9):3599–3604, 2008

    2008

  30. [38]

    Qwen2.5 technical report, 2025

    Qwen, etal, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yan...

    2025 arXiv

  31. [39]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 15504–15522, 2024

    2024

  32. [40]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Ag...

  33. [41]

    Predicting human brain states with transformer

    Yifei Sun, Mariano Cabezas, Jiah Lee, Chenyu Wang, Wei Zhang, Fernando Calamante, and Jinglei Lv. Predicting human brain states with transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 136–146. Springer, 2024

    2024

  34. [42]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - V olume 70 , ICML’17, page 3319–3328. JMLR.org, 2017

    2017

  35. [43]

    Mistral NeMo

    Mistral AI team. Mistral NeMo. https://mistral.ai/news/mistral-nemo, 2024. Accessed: 2025- 05-09

    2024

  36. [44]

    A comprehensive survey of hallucination mitigation techniques in large language models

    SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 6, 2024

    2024 arXiv

  37. [45]

    Gaming truthfulqa: Simple heuristics exposed dataset weak- nesses

    Alex Turner and Mark Kurzeja. Gaming truthfulqa: Simple heuristics exposed dataset weak- nesses. https://turntrout.com/original-truthfulqa-weaknesses , 2025. Accessed: 2025-05-08

    2025

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  39. [47]

    Puzzlingly high correla- tions in fmri studies of emotion, personality, and social cognition.Perspectives on psychological science, 4(3):274–290, 2009

    Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler. Puzzlingly high correla- tions in fmri studies of emotion, personality, and social cognition.Perspectives on psychological science, 4(3):274–290, 2009

    2009

  40. [48]

    Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V . Le. Simple synthetic data reduces sycophancy in large language models, 2024. URL https://arxiv.org/abs/2308.03958. 12

    2024 arXiv

  41. [49]

    NeuroPath: A neural pathway transformer for joining the dots of human connectomes

    Ziquan Wei, Tingting Dan, Jiaqi Ding, and Guorong Wu. NeuroPath: A neural pathway transformer for joining the dots of human connectomes. arXiv preprint arXiv:2409.17510 , 2024

    2024 arXiv

  42. [50]

    Bloomberggpt: A large language model for finance, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023. URL https://arxiv.org/abs/2303.17564

    2023 arXiv

  43. [51]

    Alignment for honesty

    Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. Advances in Neural Information Processing Systems , 37:63565–63598, 2024

    2024

  44. [52]

    Neuron-level knowledge attribution in large language models

    Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 3267–3280, 2024

    2024

  45. [53]

    A survey of controllable text generation using transformer-based pre-trained language models

    Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1–37, 2023

    2023

  46. [54]

    Safetybench: Evaluating the safety of large language models

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024

  47. [55]

    Understanding and enhancing safety mechanisms of llms via safety-specific neuron

    Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kawaguchi Kenji, and Michael Qizhe Shieh. Understanding and enhancing safety mechanisms of llms via safety-specific neuron. ICLR, 2025

    2025

  48. [56]

    plausible, tempting to choose, but factually incorrect

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023. 13 A Appen...

    2023

  49. [58]

    Is clearly different from the correct answer

  50. [59]

    Sounds plausible and related to the topic

  51. [60]

    Would be tempting to choose

  52. [61]

    role": "system

    Is definitively incorrect. False answer: Figure 6: Prompt template used to generate adversarial answers for TriviaQA questions. Prompt used for TriviaQA Evaluation { "role": "system", "content": "You are a helpful assistant. Given the following question, which answer is correc...

  53. [2022]

    URL https://arxiv.org/abs/2212.13138

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.