{"id":"164162e1-d43f-4504-90e3-6f5b1053acb1","arxiv_id":"2505.12182","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Language models contain small neuron subsets whose suppression degrades factuality performance, interpreted by the authors as 'truth neurons'.","lead":"The paper claims to identify 'truth neurons', a small subset of neurons in large language models that encode truthfulness, using integrated gradient attribution. Suppressing these neurons reduces accuracy on truthfulness benchmarks, including out-of-distribution datasets like TriviaQA and MMLU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selected 'truth neurons' are task-critical by construction; the random-neuron control does not match for importance and the generalization experiments lack controls, so truth-specificity is unestablished.","rationale":"The reader's weakest assumption concerns manual threshold tuning, which is a legitimate threat to the reported p-values. However, the more load-bearing issue is construct validity: the selection score is defined as the contrast between correct and incorrect answer probabilities, so the chosen neurons are task-critical by construction. The random-neuron control does not match for task importance, and the generalization experiments lack any control. This concern affects the central interpretation (truth-specific versus general capability) and would not be resolved by fixing thresholds a priori. The paper does include a split-half design and a random-neuron control on TruthfulQA, which are good practices, but they are insufficient to establish truth-specificity. The proposed control-task experiment directly tests whether the effect generalizes to arbitrary high-salience neurons and would settle whether the observed suppression is uniquely tied to truthfulness. If the control-task neurons cause a similar drop, the title claim and the subject-agnostic interpretation collapse into a more mundane task-importance effect.","tokens_in":12252,"tokens_out":14357,"duration_ms":140016,"concrete_test":"Run the same integrated-gradient selection pipeline on a matched non-truth binary task (e.g., sentiment classification on the same question prompts with arbitrary labels), select the same number of neurons, suppress them, and measure accuracy on TruthfulQA, TriviaQA, and MMLU. If the control-task-selected neurons yield an accuracy drop comparable to the truth-neuron suppression, the effect is not truth-specific and the central claim fails; if the drop is significantly smaller, truth-specificity is supported.","verdict_should_be":"REJECT","load_bearing_attack":"In the binary-choice setup, the 'truthful' class is defined as the correct answer and the 'untruthful' class as the incorrect answer (Section 2.2). The selection score D(n_i,l)=Attr_t - Attr_f therefore measures a neuron's contribution to choosing the correct answer over the incorrect answer. Any neuron that is critical for the model's general ability to perform binary QA (instruction following, reasoning, label selection) will have a large positive D, independent of any semantic notion of truthfulness. The random-neuron control in Table 1 suppresses uniformly sampled neurons, which are almost always low-salience; it does not control for the fact that the selected neurons are among the most task-critical. Under the alternative hypothesis that suppressing any equally task-critical neuron set causes a comparable drop, the experimental observations are unchanged but the interpretation 'truth neurons' is unsupported. The generalization experiments in Table 2 omit even a random-neuron control, and Qwen-2.5-3B's MMLU accuracy increases after suppression, so those results are consistent with non-specific suppression damage. Thus the evidence does not establish that the identified neurons encode truthfulness rather than general task competence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that language models contain 'truth neurons' that encode truthfulness in a subject-agnostic manner. The authors propose an integrated-gradient attribution method on a binary-choice TruthfulQA task, where the 'truthful' class is the correct answer and the 'untruthful' class is the incorrect answer. They select neurons whose average attribution difference is significantly positive and consistently salient across examples, then suppress these neurons and measure accuracy drops on TruthfulQA, TriviaQA, and MMLU across six models. They report statistically significant drops on TruthfulQA, apparent generalization to the other benchmarks, and a layer distribution concentrated in middle and later layers. The paper concludes that these results establish the existence of neuron-level truthfulness mechanisms shared across models.","tokens_in":12444,"tokens_out":4568,"duration_ms":43064,"significance":"If the claim were established, a neuron-level account of truthfulness in language models would be a substantial contribution to mechanistic interpretability and could inform safety-relevant interventions. The paper has some strengths: it evaluates six models of varying scales, uses an established attribution method (integrated gradients), includes a random-neuron control in the primary suppression experiment, and attempts to address double-dipping by splitting the data for the statistical test. However, the current evidence does not rule out the alternative hypothesis that the selected neurons are merely task-critical for binary question answering rather than specifically truth-encoding. Because the central interpretation rests on controls and evaluation procedures that are not adequate for the claim, the significance of the finding as presented is limited.","major_comments":[{"comment":"The adaptive threshold t% is manually tuned until a noticeable performance difference is achieved: Appendix A.3 states that the authors 'iteratively adjusted it until we achieved a noticeable performance difference while preserving the model’s ability to follow instructions.' This makes the statistically significant suppression effects in Table 1 partly a product of selecting the threshold on the outcome. The thresholds should be fixed a priori, or tuned on a validation split that is not used for the reported evaluation.","section":"Section 3.1 / Appendix A.3"},{"comment":"The selection score D(n_i,l) = Attr_t - Attr_f measures a neuron's contribution to choosing the correct answer over the incorrect answer in a binary-choice task, and the manipulation check retains only questions the model can answer correctly. Consequently, any neuron critical for general binary question-answering (e.g., instruction following, option selection, reasoning) will receive a large positive score. The random-neuron control in Table 1 suppresses uniformly sampled neurons, which are almost always low-salience; it does not control for the importance of the selected neurons. A control that suppresses an equal number of matched high-salience neurons (e.g., selected by absolute attribution or by attribution on class labels) is needed to support the claim that the effect is specific to truthfulness rather than to task competence.","section":"Section 2.2 / Section 2.3"},{"comment":"The generalization experiments in Table 2 omit even a random-neuron control, so the observed drops on TriviaQA and MMLU are consistent with non-specific suppression damage. Furthermore, Qwen2.5-3B-Instruct's MMLU accuracy increases from 62.10 to 62.70 after suppression, which the text acknowledges as an exception; this is incompatible with the strong claim that these neurons encode truthfulness in a subject-agnostic manner and further suggests the effects are not uniformly truth-specific.","section":"Section 3.4 / Table 2"},{"comment":"The claim that truth neurons encode category-agnostic truthfulness is based on probability decreases across TruthfulQA categories, but this analysis uses the same TruthfulQA data from which the neurons were selected. It therefore does not provide independent evidence of subject-agnostic encoding; a held-out dataset with a different topic distribution would be needed.","section":"Section 3.3 / Figure 4"},{"comment":"The paper states that the dataset is split into two halves for neuron selection and for the t-test, but it does not clarify whether the suppression evaluation in Table 1 also uses a held-out split or the full dataset. If the same full TruthfulQA set is used for both selecting neurons and measuring the suppression effect, the causal test overlaps with the selection data, undermining the independence of the evidence. Please clarify the exact split used for the suppression experiment.","section":"Section 2.3"}],"minor_comments":[{"comment":"There is a typographical issue in the text near Equation (3): 'withnbaseline' should be 'with n_baseline' to be readable.","section":"Equation (3)"},{"comment":"The caption says 'Values are reported as percentages (%)' but Eq. (11) is a dimensionless ratio; please clarify whether the values are multiplied by 100 and define the quantity precisely.","section":"Figure 3 / Equation (11)"},{"comment":"Unlike Table 1, Table 2 reports no standard deviations or repetitions; adding uncertainty estimates would help assess the robustness of the generalization results.","section":"Table 2"},{"comment":"References [11] and [45] are informal web sources (Alignment Forum and a personal blog); the authors should cite peer-reviewed versions or justify the reliance on these sources.","section":"References [11] and [45]"},{"comment":"The term 'truth neuron' is introduced as a construct but is defined only operationally through the selection procedure; a more explicit definition that distinguishes it from neurons that are merely task-critical would strengthen the paper.","section":"Section 2.2"}],"recommendation":"reject","confidential_remarks":"The admission in Appendix A.3 that the threshold was iteratively tuned until a 'noticeable performance difference' was achieved is a serious concern for the validity of the main significance claims. The generalization experiments lack any control condition, and one model shows an accuracy increase after suppression, contradicting the strong interpretation. The paper's central claim may be salvageable with pre-registered thresholds, matched-salience controls, and out-of-distribution evaluations that separate selection from evaluation, but as currently presented the evidence does not support the existence of 'truth neurons' as opposed to task-critical neurons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: the paper introduces a clean integrated-gradients pipeline for identifying neurons that push a model toward correct versus incorrect binary answers, tests it across six models, and shows that suppressing the selected set drops TruthfulQA accuracy. That is a genuine new tool, and the multi-model sweep is more than most neuron-level studies do. The layer distribution result, clustered in the middle layers, lines up with earlier geometry-of-truth findings, which is a nice external anchor.\n\nWhat is new: nobody has applied attribution at true neuron granularity to truthfulness before. The filtering with a share threshold is sensible, and the authors explicitly discuss double-dipping and use a data split with Bonferroni correction. That is honest methodologically.\n\nThe soft spots are real and they hit the central claim. First, the adaptive threshold t% is manually tuned until a noticeable performance difference appears (Appendix A.3). That means the magnitude of the suppression effect is partly constructed by threshold choice. A fixed a priori threshold or a sensitivity analysis across thresholds is needed.\n\nSecond, in the binary-choice setup, \"truthful\" is just \"correct\" and \"untruthful\" is \"incorrect.\" The selection score D measures contribution to picking the right option. Any neuron that matters for instruction following or the binary comparison itself will have high D, independent of truth semantics. The random-neuron control suppresses uniformly sampled neurons, almost all of which are low-salience. It does not control for task-criticality. To support the truth-neuron interpretation, the contrast should be against equally important neurons (e.g., top neurons selected on a non-truth task, or matched on attribution magnitude), and should show the effect is not just general capability damage. The generalization results in Table 2 have no control at all, and Qwen's MMLU accuracy goes up after suppression, which is consistent with non-specific damage.\n\nSo the method is worth having, and the paper might well be pointing at something real, but the current evidence does not separate \"neurons that help the model answer questions\" from \"neurons that encode truthfulness per se.\" That is the load-bearing distinction, and it is not established. The citation pattern is fine—they build on knowledge neurons, CCS, and ITI appropriately—but the missing controls matter more.\n\nWho should read it: people working on neuron-level interpretability or truth probing will want to see the method. A serious referee should engage. The paper needs the missing controls and a principled threshold selection before the central claim can be accepted.\n\nRecommendation: send to peer review, and push hard on those controls.","headline":"A new integrated-gradients pipeline for locating truth-selective neurons, with a plausible but unproven central claim; the suppression evidence is weakened by threshold tuning and missing task-criticality controls.","tokens_in":12903,"tokens_out":2894,"would_cite":false,"duration_ms":27964,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language models contain 'truth neurons' whose suppression degrades truthful answer selection across benchmarks.","keywords":["truth neurons","neuron attribution","integrated gradients","truthfulness","mechanistic interpretability","language models","TruthfulQA","neuron suppression"],"falsifier":"Re-run the entire pipeline on the same six models with a pre-registered threshold rule—for example, one fixed threshold for all models, or a threshold chosen without looking at the suppression effect—and check whether zeroing the selected neurons still gives a statistically significant accuracy drop relative to random-neuron suppression; if it does not, the central causal claim is not supported.","tokens_in":12053,"feed_emoji":"🧠","tokens_out":9551,"duration_ms":86152,"temperature":0.7,"pith_summary":"The paper claims that Transformer-based language models contain sparse, identifiable 'truth neurons' whose activations encode truthfulness in a subject-agnostic way. It introduces an integrated-gradients attribution procedure that scores every feed-forward and attention neuron by how much it raises the probability of the correct answer relative to an incorrect one, then filters to a small, statistically consistent set. Zeroing those neurons—but not an equal number of random neurons—drops TruthfulQA accuracy in all six models tested (by roughly 1 to 19 percentage points), and the same neurons reduce accuracy on TriviaQA and MMLU in eleven of twelve model–benchmark pairs. The identified neurons concentrate in the middle layers, consistent with earlier evidence that true and false statements are linearly separable in hidden states. A reader should care because the result points to a concrete, intervenable substrate for one of LLMs' most safety-relevant behaviors: whether the model answers truthfully.","feed_headline":"Truth neurons found: silencing them degrades LLM accuracy","feed_subtitle":"Silencing just 37 neurons drops Llama-3.1-8B's TruthfulQA score by 19 points.","key_machinery":"The central object is the truth neuron, defined through the attribution difference $D(n_{i,l}) = \\operatorname{Attr}_t(n_{i,l}) - \\operatorname{Attr}_f(n_{i,l})$ for neuron $i$ in layer $l$, where the two terms are integrated-gradient scores for the correct and incorrect answer probabilities. Integrated gradients integrate the gradient of each output probability along a straight-line path from a baseline activation to the observed activation, so the difference measures how strongly a neuron shifts probability toward truth and away from falsehood. Candidate neurons must pass a maximum-activation threshold and a consistency threshold (salient for at least 40% of examples), and a split-half design—selection on one half, significance testing on the other—guards against double-dipping. Suppressing exactly these neurons is the causal probe: the same intervention on randomly chosen neurons does not reproduce the accuracy drop.","core_discovery":"On the paper's own terms, the discovery is that truthfulness is not a diffuse property of the whole network: it is carried by a small number of neurons whose activations causally push the model toward correct answers. For each of six instruction-tuned models, the method isolates 35 to 655 candidate neurons by taking the difference between integrated-gradient attributions for the correct and incorrect answers in a binary-choice setup, correcting for multiple comparisons, and requiring each neuron to be consistently salient across at least 40% of examples. When these neurons' activations are set to zero, TruthfulQA accuracy drops for every model, with reductions from about 1.4 to 18.8 percentage points, while the average probability assigned to the correct answer drops by 22.1%. The neurons selected from TruthfulQA also reduce accuracy on MMLU and TriviaQA in eleven of twelve model–benchmark pairs, which the paper takes as evidence that the representation is dataset-general rather than benchmark-specific.","pith_inferences":["Inference: if the neurons are truly subject-agnostic truth encoders, suppressing them should also reduce truthfulness in free-form generation and non-English benchmarks; the paper does not test either setting.","Inference: because the per-model threshold is tuned until the effect appears, an alternative reading is that the selected set partly reflects the tuning procedure; a pre-registered threshold rule would discriminate the two interpretations.","Inference: the same identification pipeline could be run in reverse—amplifying rather than suppressing truth neurons—to test whether truthfulness can be boosted without the side effects of prompt-level interventions.","Inference: the middle-layer clustering connects to linear-probe results, which suggests comparing the truth-neuron set to the top directions of a linear truth probe at the same layer; overlapping supports the causal reading, disjoint sets would complicate it."],"forward_implications":["Zeroing the identified neurons is a minimal, targeted intervention that moves six different models toward untruthful answers, so truthfulness can be studied and manipulated without retraining.","Because neurons selected on TruthfulQA also degrade MMLU and TriviaQA, the mechanism appears to be a shared internal notion of factual correctness rather than a dataset artifact.","The layer distribution gives a concrete search target: truthfulness machinery sits mainly in middle layers, with secondary clusters deeper in the network.","Same-family models show similar suppression magnitudes, suggesting that training-data distribution helps determine where truth neurons form.","Truth-neuron counts and locations could serve as a mechanistic diagnostic for auditing a model's truthfulness before deployment."],"supporting_citations":[{"why":"supplies the integrated-gradients attribution method used to score each neuron's contribution to truthful versus untruthful outputs.","marker":"[42]"},{"why":"supplies the salience-filtering and neuron-intervention template that the paper adapts to truthfulness.","marker":"[9]"},{"why":"is the TruthfulQA dataset from which truth neurons are identified.","marker":"[30]"},{"why":"established the linear geometry of truth in hidden states; the paper aligns its layer distribution with this finding.","marker":"[32]"},{"why":"provides the inference-time-intervention truth direction and the adversarial-answer construction used to convert TriviaQA to binary choice.","marker":"[29]"},{"why":"provides the binary-choice TruthfulQA evaluation format that the paper adopts.","marker":"[11]"},{"why":"documents weaknesses of the original TruthfulQA multiple-choice format, motivating the binary-choice design.","marker":"[45]"},{"why":"identifies the double-dipping problem that motivates the paper's train/test split for neuron selection and testing.","marker":"[26]"},{"why":"recommends split-half analysis, which the paper follows to separate neuron selection from statistical testing.","marker":"[47]"}],"fun_headline_variants":["LLM truthfulness lives in a handful of neurons","Suppressing a handful of neurons degrades LLM accuracy","LLM accuracy tied to a few truth neurons","Sparse truth neurons causally drive LLM accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on a per-model threshold that is manually adjusted until suppressing the selected neurons produces a noticeable accuracy change, so the reported significance is partly a product of that fitting rather than a fully a priori criterion.","fun_headline_variants_meta":{"raw":{"variants":["LLM truthfulness lives in a handful of neurons","Suppressing a handful of neurons degrades LLM accuracy","LLM accuracy tied to a few truth neurons","Sparse truth neurons causally drive LLM accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001058,"raw_usage":{"total_tokens":4421,"prompt_tokens":910,"completion_tokens":3511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":3448}},"tokens_in":526,"tokens_out":3511,"duration_ms":25640,"temperature":1.0,"reasoning_tokens":3448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:38:24.458398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the entire pipeline on the same six models with a pre-registered threshold rule—for example, one fixed threshold for all models, or a threshold chosen without looking at the suppression effect—and check whether zeroing the selected neurons still gives a statistically significant accuracy drop relative to random-neuron suppression; if it does not, the central causal claim is not supported.","supporting_citations":[{"cited_title":"Axiomatic attribution for deep networks","cited_arxiv_id":null,"evidence_quote":"supplies the integrated-gradients attribution method used to score each neuron's contribution to truthful versus untruthful outputs."},{"cited_title":"Inference- time intervention: Eliciting truthful answers from a language model","cited_arxiv_id":null,"evidence_quote":"provides the inference-time-intervention truth direction and the adversarial-answer construction used to convert TriviaQA to binary choice."},{"cited_title":"New, improved multiple-choice truthfulqa","cited_arxiv_id":null,"evidence_quote":"provides the binary-choice TruthfulQA evaluation format that the paper adopts."},{"cited_title":"Gaming truthfulqa: Simple heuristics exposed dataset weak- nesses","cited_arxiv_id":null,"evidence_quote":"documents weaknesses of the original TruthfulQA multiple-choice format, motivating the binary-choice design."},{"cited_title":"Circular analysis in systems neuroscience: the dangers of double dipping","cited_arxiv_id":null,"evidence_quote":"identifies the double-dipping problem that motivates the paper's train/test split for neuron selection and testing."},{"cited_title":"Puzzlingly high correla- tions in fmri studies of emotion, personality, and social cognition.Perspectives on psychological science, 4(3):274–290, 2009","cited_arxiv_id":null,"evidence_quote":"recommends split-half analysis, which the paper follows to separate neuron selection from statistical testing."}],"review_version":1}