Pith. sign in

REVIEW 2 major objections 5 minor 43 references

The Geometry of Self-Verification in a Task-Specific Reasoning Model

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Disabling three attention heads in a CountDown reasoning model makes it reject its own correct solutions and keep searching indefinitely.

desk verdict A genuinely useful causal localization technique, but the headline 'three heads' claim is an in-sample optimum, not a stable circuit. read the letter →

arxiv 2504.14379 v2 pith:J2K3B3F3 submitted 2025-04-19 cs.AI cs.LG

Andrew Lee , Lihao Sun , Chris Wendler , Fernanda Viégas , Martin Wattenberg This is my paper · ORCID
classification cs.AIcs.LG
keywords self-verificationchain-of-thoughtmechanisticinterpretabilityattentionheadsGLUCountDowntaskreasoningmodelscausalintervention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that self-verification in a reasoning model is not a diffuse side effect of training but a localizable mechanism. The authors train a CountDown solver with the DeepSeek R1 reinforcement-learning recipe, which mode-collapses the model into producing highly structured chain-of-thought traces where each arithmetic attempt is explicitly marked as '(this works)' or '(not {ans})'. Using linear probes on one side and causal head ablation on the other, they converge on a small set of 'previous-token' attention heads that carry the hidden state into a GLU-defined verification subspace; switching off just three of them (L17H14, L17H11, L17H10) makes the model label correct solutions as invalid and continue generating indefinitely. If the claim holds, it matters because verification can be read and manipulated in hidden states rather than only in the textual chain of thought, and the same components appear in the base model and in a larger general reasoning model.

What carries the argument

The load-bearing object is the verification polytope $S_{\text{GLUValid}}$, defined as the intersection of the receptive fields of the GLUValid neurons: each receptive field is the set of hidden states for which a GLU's gating-and-up projection is positive. GLUOut vectors are the rows of the GLU output matrix, so a GLU block emits a weighted sum of these vectors, and the paper selects the ones whose directions best match a linear probe separating correct from incorrect attempts. The selected vectors promote tokens such as 'this', 'success', and 'yes', while their antipodes promote 'not' and negative Chinese vocabulary. The bridge between attention and GLUs is a composition score (Eq. 8) that measures how strongly each previous-token head's OV circuit activates the gate and up weights of the GLUValid set; this score is what narrows the field to the three heads. A secondary mechanism is the SiLU nonlinearity: inactive GLUInvalid neurons take small negative values, which can flip their contribution toward the success direction, explaining why ablating GLUInvalid as well as GLUValid improves intervention.

What would settle it

Select the attention heads on one half of a fresh set of CountDown solutions and evaluate them only on the other half; if the intervention success rate on held-out examples falls to the random-baseline level, the 'as few as three' localization is an artifact of selection rather than a stable verification circuit.

Watch

Extended reading notes

Core claim

The paper claims that self-verification in its trained CountDown model is handled by a small set of 'previous-token' attention heads that direct the hidden state into a verification subspace carved out by GLU gating and up-projection weights. Zeroing the output weights of three such heads, L17H14, L17H11, and L17H10, makes the model label an already-correct arithmetic attempt as '(not {ans})' and continue its chain of thought without terminating; the same operation collapses activation of the verification-related GLU vectors. The paper further claims that the same mechanism, with more redundancy, exists in the base model and in DeepSeek-R1-Distill-Qwen-14B, where the intervention first mislabels a correct answer but the model later corrects itself. These experiments point to a necessary component of a larger verification circuit rather than the whole circuit.

Load-bearing premise

The head-selection procedure and the intervention success measurement use the same 300 samples, so the near-perfect 'as few as three heads' result may reflect selection on those examples rather than a property that generalizes to new examples.

Editorial extensions

If this is right

  • Self-verification is separable from arithmetic competence: with the three heads off, the model still computes correct expressions but no longer recognizes them as valid.
  • The verification signal is linearly readable and steerable: adding the probe direction makes the model declare a wrong attempt correct.
  • The GLU dictionary is semantically organized around verification, with valid/invalid vectors and their antipodes aligning with success and negation tokens across English and Chinese.
  • The mechanism predates RL: the base model already uses previous-token heads for verification, and RL appears to concentrate the required computation into fewer heads.
  • Larger reasoning models keep more redundancy: in the 14B model the same interventions cause temporary misclassification followed by self-correction, suggesting a larger verification circuit than three heads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the same localization scales to larger production models, a small set of heads could serve as a hidden-state monitor for whether the model believes it has solved a problem, bypassing the disputed faithfulness of chain-of-thought text.
  • Editorial inference: the three-head set is likely a lower bound rather than the circuit; Appendix F already shows that other scoring choices need anywhere from 8 to 100 heads, so the mechanism is a family of near-equivalent routes, not one unique wire.
  • Editorial inference: the SiLU-antipode account makes a concrete prediction—replacing SiLU with ReLU in the GLU blocks should remove the advantage of also ablating GLUInvalid vectors.
  • Editorial inference: because head selection and evaluation share the same 300 samples, a clean test of the central claim is to pick heads on one batch of CountDown problems and measure intervention success only on a fresh batch; the paper does not report that split.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reverse-engineers self-verification in a task-specific CountDown reasoning model trained with DeepSeek R1-Zero's recipe. It combines top-down probing of GLU output vectors, bottom-up identification of 'previous-token' attention heads, and a composition score (Eq. 8) to localize three attention heads (L17H14, L17H11, L17H10) whose ablation disables verification, causing the model to misclassify correct solutions as invalid and continue its chain of thought indefinitely. The authors validate analogous components in the base Qwen2.5-3B model and in DeepSeek-R1-Distill-Qwen-14B via in-context learning and EMB2EMB transfer. The paper claims these heads are a 'necessary component of a potentially larger verification circuit,' not a full circuit.

Significance. If the central localization claim were robust, this would be a valuable contribution to mechanistic interpretability of reasoning models: it would show that a concrete, causally testable verification mechanism is mediated by a small set of attention heads and that similar components appear across model sizes. Strengths include the deliberate use of mode collapse to create a structured, parseable chain of thought; the combination of linear probes, GLU decomposition, and causal interventions; and the cross-model validation in a base model and a 14B reasoning model. The paper is also appropriately cautious in disclaiming a full circuit and in noting the task-specific scope. However, the headline 'as few as three heads' result is not yet supported by a held-out evaluation, which is the main obstacle to accepting the central claim as stated.

major comments (2)
  1. [Section 4.3, Section 4.4, Appendix F] The 'as few as three heads' claim is selected and evaluated on the same 300-sample test set. Section 4.3 states that heads are ablated 'one head at a time until we achieve perfect intervention scores,' and Section 4.4 reports intervention success on the same 300 samples. Because the selection objective is the intervention success rate on that set, the minimal head count is an in-sample optimum, not an out-of-sample prediction. Appendix F confirms the instability: varying the hyperparameter N in Eq. 8 changes the required number of heads from 3 (N=200) to 15 (N=50), 100 (N=100), and 12 (N=300), and the head identities differ substantially across methods. The random-baseline comparison in Figure 2 is also biased because the selected heads were explicitly chosen to beat that baseline on the same examples. To support the central claim, the authors should either split the selection and evaluation sets (e.g., select on a training set and evaluate on a held-out set, or use cross-validation) or explicitly characterize the three-head result as an in-sample descriptive finding and provide a stability analysis across resampled test sets. Without such an analysis, the memorable localization of 'as few as three heads' is not supported.
  2. [Equation (8), Appendix F, Section 4.3] The composition score in Eq. (8) is acknowledged to ignore head interactions, layer norms, and intermediate nonlinearities, but it is the sole ranking criterion for the greedy ablation. The large variation in selected head sets across values of N and across alternative scoring methods (Table 6) suggests that the three-head set is not a stable, unique circuit component but rather one of many possible subsets that can disrupt verification when ablated. The paper would be strengthened by a robustness analysis that reports, for each head set in Table 6, the corresponding intervention success rate and overlap with the proposed three-head set. As written, the claim that these three specific heads form 'a necessary component' is disproportionately stronger than what the evidence shows, given that many other subsets of comparable or larger size achieve the same effect.
minor comments (5)
  1. [Appendix D, line 2] The phrase 'incorect CoT tokens' contains a typo; it should read 'incorrect CoT tokens.'
  2. [Section 6, paragraph 1] The sentence 'A growing line or work decomposes the forward pass' contains a typo; 'or' should be 'of.'
  3. [Figure 2 caption] The caption contains the word 'samping' in the description of the AVerif baseline; it should be 'sampling.'
  4. [Section 2, notation] The notation 'xℓ∈ I Rd,ℓ∈ [L− 1]' is confusing because [L−1] is not defined as a set of integer indices; clarify that ℓ ranges over layer indices 0,...,L−1.
  5. [Abstract, Section 4.3] The phrase 'as few as three attention heads' is used without the caveat that this number depends on the choice of N in Eq. (8) and on the selection procedure. Given Appendix F, the abstract should qualify the claim (e.g., 'with the default hyperparameters, ...') to avoid overstatement.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'as few as three heads' localization is an in-sample optimum: heads are greedily selected on the same 300-sample test set used to report near-perfect intervention success.

  1. fitted input called prediction [Section 4.3 (head selection) and Section 4.4 (intervention test set), with Appendix F (Table 6).]
    "Once we score each previous-token head using Eq. 8, we incrementally ablate one head at a time until we achieve perfect intervention scores (Section 4.4). Using this approach, we identify as few as three attention heads that can disable model verification. We notate this subset as AVerif."

    Section 4.3's greedy rule is 'until we achieve perfect intervention scores', and Section 4.4 measures 'intervention success rate' on 'our test set consists of 300 samples'. Thus AVerif's near-perfect success is the optimized metric evaluated on the same examples used for selection; the procedure stops only when a set scores perfectly on that set. No held-out evaluation is reported. Appendix F/Table 6 shows instability: Eq. 8 with N=200 uses 3 heads, N=50 uses 15, N=100 uses 100, N=300 uses 12, and alternative scorers use 8-17. The 'as few as three' count is therefore an in-sample optimum, not a stable out-of-sample circuit. The random-head baseline is also biased because the selected heads were chosen to beat it on the same 300 examples.

full rationale

The paper's core derivation is not globally circular: the probe W is fit to xValid/xInvalid, and the GLUValid vectors are chosen by similarity to W, but the decisive evidence for these components is causal (zeroing heads/GLU weights changes behavioral tokens), which is not implied by the linear-algebra selection. The self-cited EMB2EMB method [16] is used only to transfer the probe to R1-14B; it is a described least-squares map, not an unverified uniqueness theorem, so it does not load-bearingly reduce the R1-14B finding to the authors' prior claim. However, the headline 'as few as three attention heads' claim is circular in the narrower sense of fitted-input-called-prediction: the head subset AVerif is greedily selected until perfect 'intervention scores' on the same 300-sample test set on which the success rate is then reported (Sections 4.3-4.4). The stopping rule makes a perfect-scoring subset exist by construction on that set, and Appendix F shows the minimal count varies from 3 to 100 with hyperparameter N, confirming the number is selected, not predicted. The paper's own limitation statement that this is 'a necessary component of a potentially larger verification circuit' is appropriate but does not repair the in-sample nature of the three-head localization.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The pipeline rests on several fitted or hand-selected ingredients: a linear probe trained on hidden states, a threshold for previous-token heads, a top-k GLU selection, and an N value in the composition score that directly controls how many heads are found. The causal ablation is the strongest evidence, but the specific three-head result is selected on the evaluation set, so the ledger reflects that internal tuning.

free parameters (5)
  • Probe W (linear classifier) = Per-layer W^l in R^{2xd}, trained on 327,680 samples
    Identifies the linear direction separating x_valid from x_invalid; all downstream GLUValid/GLUInvalid selection depends on it.
  • Top-k GLU vectors per layer = 50 per layer per direction
    Number of GLUOut vectors selected by cosine similarity to the probe; this choice affects which GLU vectors and later heads are identified.
  • Previous-token attention threshold = 10% for R1CountDown; 5% for R114B
    Defines the APrev set; 10% yields 33 heads in the main model, while 5% yields 552 heads in R114B and 10% gives much lower intervention success there.
  • N in composition score (Eq. 8) = 200 in main text
    Number of GLUValid vectors used to score attention heads. Appendix F shows varying N from 50 to 300 changes the minimal selected head set from 3 to 100, so the 'as few as three' result is sensitive to this hyperparameter.
  • EMB2EMB transformation T = Learned linear map between Qwen2.5-3B and R1-14B embedding spaces using 100,000 tokens
    Used to transfer the probe to R114B; validity depends on cross-model linearity of embedding spaces.
assumptions (5)
  • domain assumption A linear direction in activations (probe W) captures the model's verification state.
    The paper trains linear probes and reports >90% validation accuracy, but this assumes the relevant computation is linearly decodable rather than distributed in a nonlinear way.
  • domain assumption The CoT tokens 'this works' and 'not {ans}' are reliable behavioral labels for internal verification states.
    Ground-truth labels for probe training and intervention measurement are derived from these tokens; if the tokens are stylistic artifacts of mode collapse rather than tied to correctness computation, the analysis measures token prediction, not verification.
  • ad hoc to paper The OV circuit composition score (Eq. 8) approximates the causal effect of an attention head on GLUValid neurons.
    Equation 8 ignores layer norms, attention distributions, residual connections, and cross-head interactions; the authors acknowledge these simplifications in Section 4.3, yet the score is the basis for localizing the three heads.
  • domain assumption Nearest-neighbor tokens in the unembedding space reveal semantic content of GLU vectors.
    Table 2 interprets GLUValid/GLUInvalid by embedding-space neighbors; this assumes token embedding cosine similarity reflects semantic similarity.
  • domain assumption EMB2EMB transfers steering directions between models with different architectures and vocabularies.
    Used to define a probe vector for R114B; transfers a linear map fitted on 100,000 random tokens across embedding spaces.
invented entities (1)
  • Verification subspace (polytope) S_GLUValid
    purpose: A hypothesized activation subspace that the model's hidden state enters when it has verified its solution; activating it promotes verification tokens like 'success'.
    The subspace is inferred from probe directions and GLU activations; the causal evidence (head ablation reduces GLUValid activations and disables verification) is internal to the model, not an external falsifiable handle such as a measurable prediction outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Geometry of Self-Verification in a Task-Specific Reasoning Model." pith.science (2026). https://pith.science/paper/J2K3B3F3

@misc{pith2026250414379,
  author       = {Pith},
  title        = {Pith review of: The Geometry of Self-Verification in a Task-Specific Reasoning Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2K3B3F3}},
  note         = {Machine review of arXiv:2504.14379}
}
read the original abstract

How do reasoning models verify their own answers? We study this question by training a model using DeepSeek R1's recipe on the CountDown task. We leverage the fact that preference tuning leads to mode collapse, yielding a model that always produces highly structured chain-of-thought sequences. With this setup, we do top-down and bottom-up analyses to reverse-engineer how the model verifies its outputs. Top-down, we find Gated Linear Unit (GLU) weights encoding verification-related tokens, such as ``success'' or ``incorrect''. Bottom-up, we find that ``previous-token heads'' are mainly responsible for self-verification in our setup. Our analyses meet in the middle: drawing inspiration from inter-layer communication channels, we use the identified GLU weights to localize as few as three attention heads that can disable self-verification, pointing to a necessary component of a potentially larger verification circuit. Finally, we verify that similar verification components exist in our base model and a general reasoning DeepSeek-R1 model.

Figures

Figures reproduced from arXiv: 2504.14379 by the authors.

Figure 1
Figure 1. Averaged LogitLens from 300 samples. We see tokens related to verification (“success”, “不合”) in the last few layers. (a), (b) show the top tokens when (in)correct solutions are reached. (c), (d) shows results from intervening on either GLU weights or attention heads, given a correct solution. For (c), while the model is less certain (P(“this”) drops from 0.98 to 0.54), we still see tokens such as “success” showing u… view at source ↗
Figure 2
Figure 2. Intervention Results: Disabling as few as 3 attention heads disables self-verification, rendering the model to generate tokens indefinitely. APrev refers to 33 previous-token heads. APrev Baseline refers to the average of 5 runs, each run randomly sampling 33 attention heads. AVerif refers to a subset of 3 previous-token heads. AVerif Baseline refers to the average from 5 runs, each run randomly samping 3 attention … view at source ↗
Figure 3
Figure 3. GLUValid activations before and after turning off 3 AVerif attention heads. Adjacent pairs of blue and orange bars indicate the same GLUValid vector. Turning off our identified attention heads leads to a significant drop in their activations. 4.4 Causal Interventions We study the role of each component above with causal intervention. Our test set consists of 300 samples in which the model originally correctly finds … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Intervention Results for the base model and R114B. In the base model, APrev can similarly disable self-verification, while AVerif only plays a partial role for verification, hinting at the effects of RL on their weights. In R114B, interventions mostly leads to partial …
Figure 5
Figure 5. Figure 5: Averaged LogitLens from 300 samples (Same as [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Probe Accuracy. The model has a linear separation in its hidden states given correct versus incorrect CoT tokens. Varying Hyperparameters. Given our approach (Section 4.3), we try different parameters for N. Sort by Similarity to Wgate, Wup. An alternative approach is …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 14 canonical work pages

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  2. [2]

    Chain-of-thought reasoning in the wild is not always faithful.arXiv preprint arXiv:2503.08679, 2025

    Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful.arXiv preprint arXiv:2503.08679, 2025

    arXiv 2025

  3. [3]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  4. [4]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations

    work page

  5. [5]

    Designing a dashboard for transparency and control of conversational ai

    Yida Chen, Aoyu Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, et al. Designing a dashboard for transparency and control of conversational ai. arXiv preprint arXiv:2406.07882, 2024

    arXiv 2024

  6. [6]

    Transcoders find interpretable llm feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page

  7. [7]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

  8. [8]

    Do i know this entity? knowledge awareness and hallucinations in language models

    Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257, 2024

    arXiv 2024

Show all 43 references
  1. [9]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  2. [10]

    Stream of search (sos): Learning to search in language

    Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.03683, 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 10

    2016 arXiv

  5. [13]

    Understanding the effects of rlhf on llm generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024

    2024

  6. [14]

    Measuring faithfulness in chain-of-thought reasoning

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023

    2023 arXiv

  7. [15]

    A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In International Conference on Machine Learning, pages 26361–26378. PMLR, 2024

    2024

  8. [16]

    Shared global and local geometry of language model embeddings

    Andrew Lee, Melanie Weber, Fernanda Viégas, and Martin Wattenberg. Shared global and local geometry of language model embeddings. arXiv preprint arXiv:2503.21073, 2025

    2025 arXiv

  9. [17]

    Inference- time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530, 2023

    2023

  10. [18]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  11. [19]

    Talking heads: Understanding inter-layer communication in transformer language models

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models. Advances in Neural Information Processing Systems, 37:61372–61418, 2024

    2024

  12. [20]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013

    2013 arXiv

  13. [21]

    One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity

    Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity. arXiv preprint arXiv:2411.04427, 2024

    2024 arXiv

  14. [22]

    Emergent linear representations in world models of self-supervised sequence models

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16–30, 2023

    2023

  15. [23]

    Interpreting gpt: The logit lens, 2020

    Nostalgebraist. Interpreting gpt: The logit lens, 2020

    2020

  16. [24]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. https://distill.pub/2020/circuits/zoom-in

    2020

  17. [25]

    Olshausen and David J

    Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311–3325, 1997

    1997

  18. [26]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  19. [27]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/ . Accessed: 2025-03-21

    2025

  20. [28]

    Llms know more than they show: On the intrinsic representation of llm hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707, 2024. 11

    arXiv

  21. [29]

    Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, 2024

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, 2024

    2024

  22. [30]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning

  23. [31]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024

    2024

  24. [32]

    To backtrack or not to backtrack: When sequential search limits model reasoning

    Tian Qin, David Alvarez-Melis, Samy Jelassi, and Eran Malach. To backtrack or not to backtrack: When sequential search limits model reasoning. arXiv preprint arXiv:2504.07052, 2025

    2025

  25. [33]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, 2024

    2024

  26. [34]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  27. [35]

    Diverse preference learn- ing for capabilities and alignment

    Stewart Slocum, Asher Parker-Sartori, and Dylan Hadfield-Menell. Diverse preference learn- ing for capabilities and alignment. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  28. [36]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  29. [37]

    Understanding reasoning in thinking language models via steering vectors

    Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. In Workshop on Reasoning and Planning for Large Language Models, 2025

    2025

  30. [38]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  31. [39]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  32. [40]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  33. [41]

    Reasoning models know when they’re right: Probing hidden states for self-verification

    Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they’re right: Probing hidden states for self-verification. arXiv preprint arXiv:2504.05419, 2025

    2025 arXiv

  34. [42]

    this") Orig LogitLens (

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. 12 A Hyperparameters ...

    arXiv

  35. [2021]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.