{"id":"fa544944-b3f4-4a2f-ae19-44ffb1e1e58c","arxiv_id":"2504.14379","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In a CountDown reasoning model, three attention heads drive a verification subspace that, when disabled, makes the model mark correct solutions as incorrect.","lead":"The authors trained a small AI model on an arithmetic game and mapped the internal signals it uses to declare an answer correct. They found that disabling as few as three specific components makes the model abandon correct answers, revealing a piece of its self-verification machinery.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'as few as three heads' claim is selected on the same 300-sample test set used for evaluation; Appendix F's 3-to-100 variation across N indicates in-sample overfitting rather than a stable circuit.","rationale":"The paper's strongest evidence is causal: zeroing L17H14/L17H11/L17H10 changes behavior, and Figure 3 shows a large drop in GLUValid activations. However, the greedy selection is performed against the same 300 samples used to report success. This is not a stylistic issue: the objective being optimized ('perfect intervention scores') is exactly the evaluation metric, so the reported near-perfect rate is an in-sample training score. Appendix F makes the overfitting concrete: the number of heads needed is 3 at N=200 but 100 at N=100, and 15 or 12 at other N; different scoring functions give 8–17 heads. The three reported heads do recur across several configurations, which suggests a real core, but the paper provides no held-out estimate of how much of the near-perfect rate survives. The caveat in the text—'alternative hyperparameters or scoring functions can yield different subsets'—limits the claim from 'the verification circuit' to 'a critical component,' but the abstract's 'as few as three' remains the memorable result and is the part most exposed. A clean train/test split for the selection procedure is straightforward and would settle whether the localization is robust or a selection artifact. Because the reader already flagged this as the weakest assumption and set CONDITIONAL, this stress-test leaves the verdict unchanged.","tokens_in":18335,"tokens_out":4314,"duration_ms":39599,"concrete_test":"Split the 300 samples into two disjoint halves. Run the Section 4.3 greedy selection using only one half, then measure intervention success on the held-out half for the resulting 3-head set (and, for comparison, for the reported L17H14/L17H11/L17H10 set). Repeat across several random splits and report the distribution of held-out full/partial success rates against the random 3-head baseline. If the held-out success rate is not substantially above baseline, the 'as few as three' result is a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'as few as three attention heads can disable self-verification'—rests on a greedy search (Section 4.3) whose objective is the intervention success rate measured on the same 300-sample test set used for the headline result (Section 4.4). The text does not state that selection and evaluation are disjoint, and the stopping rule ('we incrementally ablate one head at a time until we achieve perfect intervention scores') explicitly optimizes on that set. This makes the minimal head count an in-sample optimum, not an out-of-sample estimate. Appendix F confirms the instability: with Eq. 8, N=200 yields 3 heads, but N=50 yields 15, N=100 yields 100, and N=300 yields 12; attention-density and similarity-based scorers yield 8–17 heads. A 3-head set achieving near-perfect success on the set used to select it is expected even if no stable 3-head circuit exists. The random-baseline comparison is also biased, because the selected heads are chosen to beat that baseline on the same examples. The paper is appropriately cautious in calling the result 'a necessary component of a potentially larger verification circuit,' but the memorable 'as few as three' localization is not yet supported by a held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reverse-engineers self-verification in a task-specific CountDown reasoning model trained with DeepSeek R1-Zero's recipe. It combines top-down probing of GLU output vectors, bottom-up identification of 'previous-token' attention heads, and a composition score (Eq. 8) to localize three attention heads (L17H14, L17H11, L17H10) whose ablation disables verification, causing the model to misclassify correct solutions as invalid and continue its chain of thought indefinitely. The authors validate analogous components in the base Qwen2.5-3B model and in DeepSeek-R1-Distill-Qwen-14B via in-context learning and EMB2EMB transfer. The paper claims these heads are a 'necessary component of a potentially larger verification circuit,' not a full circuit.","tokens_in":18609,"tokens_out":2964,"duration_ms":27351,"significance":"If the central localization claim were robust, this would be a valuable contribution to mechanistic interpretability of reasoning models: it would show that a concrete, causally testable verification mechanism is mediated by a small set of attention heads and that similar components appear across model sizes. Strengths include the deliberate use of mode collapse to create a structured, parseable chain of thought; the combination of linear probes, GLU decomposition, and causal interventions; and the cross-model validation in a base model and a 14B reasoning model. The paper is also appropriately cautious in disclaiming a full circuit and in noting the task-specific scope. However, the headline 'as few as three heads' result is not yet supported by a held-out evaluation, which is the main obstacle to accepting the central claim as stated.","major_comments":[{"comment":"The 'as few as three heads' claim is selected and evaluated on the same 300-sample test set. Section 4.3 states that heads are ablated 'one head at a time until we achieve perfect intervention scores,' and Section 4.4 reports intervention success on the same 300 samples. Because the selection objective is the intervention success rate on that set, the minimal head count is an in-sample optimum, not an out-of-sample prediction. Appendix F confirms the instability: varying the hyperparameter N in Eq. 8 changes the required number of heads from 3 (N=200) to 15 (N=50), 100 (N=100), and 12 (N=300), and the head identities differ substantially across methods. The random-baseline comparison in Figure 2 is also biased because the selected heads were explicitly chosen to beat that baseline on the same examples. To support the central claim, the authors should either split the selection and evaluation sets (e.g., select on a training set and evaluate on a held-out set, or use cross-validation) or explicitly characterize the three-head result as an in-sample descriptive finding and provide a stability analysis across resampled test sets. Without such an analysis, the memorable localization of 'as few as three heads' is not supported.","section":"Section 4.3, Section 4.4, Appendix F"},{"comment":"The composition score in Eq. (8) is acknowledged to ignore head interactions, layer norms, and intermediate nonlinearities, but it is the sole ranking criterion for the greedy ablation. The large variation in selected head sets across values of N and across alternative scoring methods (Table 6) suggests that the three-head set is not a stable, unique circuit component but rather one of many possible subsets that can disrupt verification when ablated. The paper would be strengthened by a robustness analysis that reports, for each head set in Table 6, the corresponding intervention success rate and overlap with the proposed three-head set. As written, the claim that these three specific heads form 'a necessary component' is disproportionately stronger than what the evidence shows, given that many other subsets of comparable or larger size achieve the same effect.","section":"Equation (8), Appendix F, Section 4.3"}],"minor_comments":[{"comment":"The phrase 'incorect CoT tokens' contains a typo; it should read 'incorrect CoT tokens.'","section":"Appendix D, line 2"},{"comment":"The sentence 'A growing line or work decomposes the forward pass' contains a typo; 'or' should be 'of.'","section":"Section 6, paragraph 1"},{"comment":"The caption contains the word 'samping' in the description of the AVerif baseline; it should be 'sampling.'","section":"Figure 2 caption"},{"comment":"The notation 'xℓ∈ I Rd,ℓ∈ [L− 1]' is confusing because [L−1] is not defined as a set of integer indices; clarify that ℓ ranges over layer indices 0,...,L−1.","section":"Section 2, notation"},{"comment":"The phrase 'as few as three attention heads' is used without the caveat that this number depends on the choice of N in Eq. (8) and on the selection procedure. Given Appendix F, the abstract should qualify the claim (e.g., 'with the default hyperparameters, ...') to avoid overstatement.","section":"Abstract, Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The core methodological concern — selection and evaluation on the same 300 samples — is substantial and directly affects the paper's main quantitative claim. The authors can likely address it with a straightforward experimental change (held-out selection or cross-validation) and by reframing the three-head result as one among several valid localizations. I would also encourage the editor to ensure the final version reports the stability of the head set across resamples, since the current Appendix F suggests high variability. The paper's overall direction is sound and the causal intervention framework is a strength, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a real contribution—a way to link attention-head OV circuits to downstream GLU vectors and causally test verification behavior—but the memorable 'as few as three heads' result does not survive contact with Appendix F. It is worth refereeing, but the authors need to run a held-out selection.\n\nWhat is actually new is the composition score between head OV circuits and GLU gate/up weights (Eq. 8), used to find a small set of heads that, when ablated, flips correct solutions to 'not' and makes the model loop. The paper also shows the same verification-related GLU directions appear in the base model and in R1-Distill-Qwen-14B using EMB2EMB, and that previous-token heads matter. The causal interventions are real and mostly convincing: zeroing 33 previous-token heads reliably kills verification, and the GLUValid activations drop as predicted. Credit where due: this is a concrete step for hidden-state monitoring, and the top-down/bottom-up meet-in-the-middle framing is clever.\n\nThe big problem is selection on the evaluation set. The three heads are found by greedily ablating until perfect intervention scores on the same 300-sample test set used for the headline number. That makes 'as few as three' an in-sample claim. Appendix F makes the instability concrete: changing N in Eq. 8 changes the minimal set to 15, 100, or 12 heads, and alternative scorers give 8–17. There is no stable three-head circuit. The authors are honest in the text—they say they have found 'a necessary component of a potentially larger verification circuit'—but the abstract and figures push the three-head story harder than the evidence supports. The random-head baseline is also weak because the selected heads were chosen to beat it. The fix is easy: select on a development set, then evaluate on a disjoint held-out set, and report the distribution over N.\n\nMinor issues: no code or data, which makes the precise numbers hard to check. The probe/GLU/head selection chain is self-referential, but the ablation is causal, so that is not fatal. The acknowledged simplifications in Eq. 8 are fine as long as the result is framed as method-dependent.\n\nWho is this for? People studying hidden-state monitoring, circuit analysis, and reasoning interpretability. It is a useful case study with a novel technique, but treat the 'three heads' as a demonstration of the method, not a discovered fact.\n\nRecommendation: send to peer review. The core finding—that previous-token heads feed a GLU-encoded verification subspace—is likely to hold with proper held-out evaluation. The three-head localization needs to be reframed as one of many possible subsets.","headline":"A genuinely useful causal localization technique, but the headline 'three heads' claim is an in-sample optimum, not a stable circuit.","tokens_in":19137,"tokens_out":2480,"would_cite":true,"duration_ms":22194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Disabling three attention heads in a CountDown reasoning model makes it reject its own correct solutions and keep searching indefinitely.","keywords":["self-verification","chain-of-thought","mechanistic interpretability","attention heads","GLU","CountDown task","reasoning models","causal intervention"],"falsifier":"Select the attention heads on one half of a fresh set of CountDown solutions and evaluate them only on the other half; if the intervention success rate on held-out examples falls to the random-baseline level, the 'as few as three' localization is an artifact of selection rather than a stable verification circuit.","tokens_in":18107,"feed_emoji":"🧠","tokens_out":8113,"duration_ms":68460,"temperature":0.7,"pith_summary":"This paper sets out to show that self-verification in a reasoning model is not a diffuse side effect of training but a localizable mechanism. The authors train a CountDown solver with the DeepSeek R1 reinforcement-learning recipe, which mode-collapses the model into producing highly structured chain-of-thought traces where each arithmetic attempt is explicitly marked as '(this works)' or '(not {ans})'. Using linear probes on one side and causal head ablation on the other, they converge on a small set of 'previous-token' attention heads that carry the hidden state into a GLU-defined verification subspace; switching off just three of them (L17H14, L17H11, L17H10) makes the model label correct solutions as invalid and continue generating indefinitely. If the claim holds, it matters because verification can be read and manipulated in hidden states rather than only in the textual chain of thought, and the same components appear in the base model and in a larger general reasoning model.","feed_headline":"Three attention heads control whether an AI trusts its own answer","feed_subtitle":"A CountDown solver rejects its own correct answers and keeps searching forever when three heads are disabled.","key_machinery":"The load-bearing object is the verification polytope $S_{\\text{GLUValid}}$, defined as the intersection of the receptive fields of the GLUValid neurons: each receptive field is the set of hidden states for which a GLU's gating-and-up projection is positive. GLUOut vectors are the rows of the GLU output matrix, so a GLU block emits a weighted sum of these vectors, and the paper selects the ones whose directions best match a linear probe separating correct from incorrect attempts. The selected vectors promote tokens such as 'this', 'success', and 'yes', while their antipodes promote 'not' and negative Chinese vocabulary. The bridge between attention and GLUs is a composition score (Eq. 8) that measures how strongly each previous-token head's OV circuit activates the gate and up weights of the GLUValid set; this score is what narrows the field to the three heads. A secondary mechanism is the SiLU nonlinearity: inactive GLUInvalid neurons take small negative values, which can flip their contribution toward the success direction, explaining why ablating GLUInvalid as well as GLUValid improves intervention.","core_discovery":"The paper claims that self-verification in its trained CountDown model is handled by a small set of 'previous-token' attention heads that direct the hidden state into a verification subspace carved out by GLU gating and up-projection weights. Zeroing the output weights of three such heads, L17H14, L17H11, and L17H10, makes the model label an already-correct arithmetic attempt as '(not {ans})' and continue its chain of thought without terminating; the same operation collapses activation of the verification-related GLU vectors. The paper further claims that the same mechanism, with more redundancy, exists in the base model and in DeepSeek-R1-Distill-Qwen-14B, where the intervention first mislabels a correct answer but the model later corrects itself. These experiments point to a necessary component of a larger verification circuit rather than the whole circuit.","pith_inferences":["Editorial inference: if the same localization scales to larger production models, a small set of heads could serve as a hidden-state monitor for whether the model believes it has solved a problem, bypassing the disputed faithfulness of chain-of-thought text.","Editorial inference: the three-head set is likely a lower bound rather than the circuit; Appendix F already shows that other scoring choices need anywhere from 8 to 100 heads, so the mechanism is a family of near-equivalent routes, not one unique wire.","Editorial inference: the SiLU-antipode account makes a concrete prediction—replacing SiLU with ReLU in the GLU blocks should remove the advantage of also ablating GLUInvalid vectors.","Editorial inference: because head selection and evaluation share the same 300 samples, a clean test of the central claim is to pick heads on one batch of CountDown problems and measure intervention success only on a fresh batch; the paper does not report that split."],"forward_implications":["Self-verification is separable from arithmetic competence: with the three heads off, the model still computes correct expressions but no longer recognizes them as valid.","The verification signal is linearly readable and steerable: adding the probe direction makes the model declare a wrong attempt correct.","The GLU dictionary is semantically organized around verification, with valid/invalid vectors and their antipodes aligning with success and negation tokens across English and Chinese.","The mechanism predates RL: the base model already uses previous-token heads for verification, and RL appears to concentrate the required computation into fewer heads.","Larger reasoning models keep more redundancy: in the 14B model the same interventions cause temporary misclassification followed by self-correction, suggesting a larger verification circuit than three heads."],"supporting_citations":[{"why":"Supplies the DeepSeek R1-Zero RL recipe with which the CountDown model is trained.","marker":"[11]"},{"why":"Introduces induction heads, the prior concept that the paper adapts into 'previous-token heads' for verification.","marker":"[26]"},{"why":"Supplies the GLU decomposition into GLUOut vectors and the inter-layer communication view used throughout.","marker":"[7]"},{"why":"Supplies the method of selecting GLUOut vectors by their similarity to a probe direction.","marker":"[15]"},{"why":"Supplies the composition-score formulation the paper adapts to score head-to-GLU communication.","marker":"[19]"},{"why":"Supplies the LogitLens method used to surface verification tokens such as 'success' and 'not'.","marker":"[23]"},{"why":"Supplies EMB2EMB, used to transfer the probe vector from the small model to the 14B model.","marker":"[16]"},{"why":"Supplies the CountDown task and the demonstration that reasoning abilities can already exist in base models before RL.","marker":"[9]"},{"why":"Supports the paper's explanation that RL fine-tuning enhances an existing verification mechanism rather than creating it from scratch.","marker":"[31]"}],"fun_headline_variants":["Three attention heads gate a model's self-trust","Self-doubt in AI: traced to three attention heads","Knocking out three heads makes an AI reject correct answers","Three heads decide if an AI accepts its own answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The head-selection procedure and the intervention success measurement use the same 300 samples, so the near-perfect 'as few as three heads' result may reflect selection on those examples rather than a property that generalizes to new examples.","fun_headline_variants_meta":{"raw":{"variants":["Three attention heads gate a model's self-trust","Self-doubt in AI: traced to three attention heads","Knocking out three heads makes an AI reject correct answers","Three heads decide if an AI accepts its own answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1282,"prompt_tokens":901,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":316}},"tokens_in":517,"tokens_out":381,"duration_ms":3914,"temperature":1.0,"reasoning_tokens":316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:50:06.591311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select the attention heads on one half of a fresh set of CountDown solutions and evaluate them only on the other half; if the intervention success rate on held-out examples falls to the random-baseline level, the 'as few as three' localization is an artifact of selection rather than a stable verification circuit.","supporting_citations":[{"cited_title":"A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity","cited_arxiv_id":null,"evidence_quote":"Supplies the method of selecting GLUOut vectors by their similarity to a probe direction."},{"cited_title":"Talking heads: Understanding inter-layer communication in transformer language models","cited_arxiv_id":null,"evidence_quote":"Supplies the composition-score formulation the paper adapts to score head-to-GLU communication."},{"cited_title":"Interpreting gpt: The logit lens, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the LogitLens method used to surface verification tokens such as 'success' and 'not'."},{"cited_title":"Fine-tuning enhances existing mechanisms: A case study on entity tracking","cited_arxiv_id":null,"evidence_quote":"Supports the paper's explanation that RL fine-tuning enhances an existing verification mechanism rather than creating it from scratch."}],"review_version":1}