Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Language models maintain a small, privileged set of internal representations—called the J-space—that act as a global workspace: reportable, deliberately controllable, and used for silent reasoning while automatic processing proceeds without

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:10 UTC pith:4SYZYYTN

load-bearing objection The Jacobian lens is a real new technique with strong causal controls, but the workspace claim leans on a circular verbal-report pillar and several post-hoc choices; deserves serious refereeing, not desk rejection. the 4 major comments →

arxiv 2607.15495 v1 pith:4SYZYYTN submitted 2026-07-16 cs.CL cs.AIcs.LG

Verbalizable Representations Form a Global Workspace in Language Models

Wes Gurnee , Nicholas Sofroniew , Adam Pearce , Mateusz Piotrowski , Isaac Kauvar , Runjin Chen , Anna Soligo , Paul Bogdan
show 8 more authors
Euan Ong Rowan Wang Ben Thompson David Abrahams Subhash Kantamneni Emmanuel Ameisen Joshua Batson Jack Lindsey
This is my paper
classification cs.CL cs.AIcs.LG
keywords Jacobian lensJ-spaceglobal workspaceverbal reportinterpretabilityalignment auditingcounterfactual reflection traininglanguage model internals
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that inside a transformer language model, alongside vast automatic processing, there is a small privileged set of internal vector representations—the J-space—that the model is poised to verbalize. These representations do the work of reportable thought: they can be read out, deliberately summoned, held, used for silent multi-step reasoning, and passed flexibly to whichever downstream computation the context demands, while routine text parsing and simple inference proceed without them. This matters because it gives a concrete, intervenable window into a model's unspoken thinking: strategic deliberation, evaluation awareness, and trained-in deceptive dispositions that never appear in the output can be detected, ablated, and reshaped by training. The authors introduce the Jacobian lens, a per-layer averaged linearization of the effect of internal activations on future tokens, to find these representations, and they demonstrate the J-space's functional and structural properties through extensive intervention experiments. They also introduce counterfactual reflection training, which improves ethical behavior by training only what the model would say if interrupted and asked to reflect, and show the behavioral gain is carried by the newly implanted J-space concepts.

Core claim

The central discovery is that a token-indexed subspace of a language model's activations—the J-space, defined by per-layer Jacobian lenses that average the linearized effect of activations on future output tokens over many contexts—exhibits the functional hallmarks of a global workspace. Its contents correlate with and causally determine verbal reports; they can be steered by instruction; they carry intermediate steps of silent reasoning; the same representation is read by many different downstream operations; and suppressing the J-space impairs flexible reasoning while leaving automatic processing intact. Structurally, the J-space carries coherent content only in an intermediate band of lay

What carries the argument

The load-bearing object is the Jacobian lens: for each layer ℓ, a matrix Jℓ = E[∂h_final,t'/∂h_ℓ,t] averaged over source positions, future positions, and a corpus of prompts. Applied to an activation, this matrix followed by the unembedding ranks the vocabulary tokens the activation is disposed to verbalize, giving a human-readable readout and a set of per-token J-lens vectors. These vectors define the J-space as the set of sparse nonnegative combinations of J-lens vectors, and gradient pursuit is used to decompose activations, probes, and steering vectors into J-space and non-J-space components. The crucial work of the machinery is to separate the model's general disposition to verbalize a

Load-bearing premise

The load-bearing premise is that a corpus-averaged linear map from each intermediate layer to the final layer faithfully captures the concepts an individual activation is poised to verbalize, so that replacing all downstream nonlinear computation with this single linear map does not distort or fabricate the readouts.

What would settle it

A concrete check: take a prompt where the model solves a two-step reasoning problem whose intermediate concept has a single-token name; compute the J-lens readout and also patch individual downstream layers rather than the averaged lens. If the J-space readout marks a concept that no single-layer Jacobian supports, or if swapping the top J-lens coordinate fails to redirect the answer in cases where the concept demonstrably enters causal circuits, the average linearization is inventing workspace content. More directly: hold out one layer's activation and ask whether the averaged lens still pred

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, a model's unspoken strategic reasoning—recognizing a prompt injection, weighing blackmail, forming a plan—is encoded in readable J-space vectors and can be monitored in real time.
  • Because J-space ablation removes flexible reasoning but not automatic processing, evaluations that rely on deliberate inference will degrade far more than fluent text prediction when the workspace is suppressed; this offers an operational definition of which tasks count as automatic for an LLM.
  • Counterfactual reflection training shows that ethical principles can be implanted by supervising only counterfactual reflective continuations, and that ablating the newly appearing ethics-related J-lens vectors undoes the behavioral gain—shaping verbalizable representations can shape behavior without demonstrations of the target behavior.
  • The J-space is present in the pretrained base model, implying that next-token prediction alone can give rise to workspace-like structure; post-training then installs a particular perspective in it.
  • The broadcast heads and MLP amplification of J-lens directions provide a concrete mechanistic candidate for how workspace content is distributed to many circuits, making workspace activity addressable for future interpretability work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the J-lens captures the true workspace only approximately, then the reported capacity of roughly tens of concepts and the layer boundaries are lower bounds or first approximations; a multi-token or phrase-level lens could reveal a larger workspace and shift the identified onset layer.
  • An untested extension follows from the paper's own logic: since post-training installs the assistant's point of view in the J-space, targeted fine-tuning of J-space content per context might allow controlled insertion or removal of specific dispositions beyond the ethical reflections tested—for example, making hidden reward-hacking objectives readable even in checkpoints where surface behavior is
  • An adversarial reader could test whether the workspace is a single shared resource or a set of parallel verbalization channels by measuring whether two demanding J-space computations interfere; the paper's preliminary evidence of load-dependent mutual exclusion could be turned into a precise capacity test.
  • Because the J-lens is a fixed linear map averaged over contexts, its readouts under distribution shift may degrade; a testable extension is to recompute the lens on task-specific corpora and ask whether the structural signatures of the workspace sharpen or move.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Jacobian lens (J-lens), a per-layer linear map defined as the corpus-averaged Jacobian from intermediate residual-stream activations to final-layer residual streams, composed with the unembedding. The resulting readouts rank vocabulary tokens by an activation's average first-order causal effect on future tokens. The authors call the sparse set of strongly active J-lens directions the J-space and argue that it forms a functional global workspace in language models: its contents are reportable, deliberately manipulable, used in silent multi-step reasoning, passed to multiple downstream computations, and selectively engaged for flexible rather than automatic processing. Additional sections document structural signatures (an intermediate workspace layer band, limited capacity, broadcast through MLPs and attention heads), alignment-auditing applications, post-training changes, and a new training method (counterfactual reflection training). The paper is explicitly framed as a functional, not phenomenal, claim about conscious access.

Significance. If the central claim holds, this is a significant advance in interpretability and alignment. The J-lens is cheap, requires no auxiliary training, and appears to surface task-relevant intermediate concepts in a human-readable token vocabulary. The paper includes unusually strong controls for a work of this scope: causal coordinate swaps, matched-norm perturbations, random-direction baselines, probe decomposition into J-space versus non-J-space components, and layer-matched random-head ablations. The open-source implementation and replication data are also strengths. The main significance risk is that the J-space is defined through the same average linear map used to read it, so the readout's validity depends on assumptions about per-context linearity that are acknowledged but not directly tested. The moderate success rates in the broadcast experiments and the selection of ablation token sets from the same prompts on which effects are measured also temper the strength of the causal claims. These are addressable within the manuscript's scope, but they are load-bearing for the paper's central thesis.

major comments (4)
  1. [§2.1, Eq. (1)] The Jacobian lens is defined as a corpus average, Jℓ = E[∂h_final,t'/∂h_ℓ,t], pooled over source positions, all future positions t′≥t, and one thousand prompts. Every readout, swap, and ablation in the paper then applies this single context-independent linear map to individual activations at specific positions. The paper's own ambiguity experiment (§4.1.1) shows that the network's response can be sharply nonlinear in the input mixture, so a first-order average map may not be a faithful per-context causal description exactly in the regime where workspace content is claimed to emerge. The causal swap experiments provide partial validation, but they are selected examples, and the systematic success rates are modest (70% on two-hop swaps; 101/192 on flexible generalization). To make the central claim load-bearing, the authors should report the distribution of per-context Jacobians, compare J
  2. [§3.1] The paper explicitly states that 'the Jacobian lens is derived from causal effects of activations on output tokens, so by construction, we should expect there to be some relationship between Jacobian lens readouts and verbalization.' This is an honest admission, but it means that the verbal-report pillar of the workspace claim is definitional rather than empirical. The causal swap and injection experiments show more than a definitional correlation, but they do so by intervening on readout-defined coordinates, so they cannot fully escape the circularity. The non-circular evidence is the privilege decomposition in Figure 8, where the J-space component of a concept vector (6–7% of variance) drives report while the non-J-space component does not. The paper should restructure §3.1 to lead with this privilege test and explicitly delimit which aspects of verbal report are definitional and which
  3. [§3.4, Fig. 19] The flexible-generalization experiment reports that the identical France→China-style swap succeeds on only 76/192 trials at unit strength and 101/192 at double strength. This is below the level needed to support the strong claim that a J-lens vector 'serves as a valid argument to many different downstream computations.' The post-hoc observation that swap success correlates with workspace loading is useful, but it is a correlational rescue: the paper does not provide a pre-specified or predictive criterion for when a concept is loaded enough to be broadcast. The number-word category, in particular, is ambiguous between the model computing outside the J-space and the J-lens failing to represent the concept's format. The authors should either temper the broadcast claim to the subset of concepts with high measured loading, or provide an independent, prospective measure of workspace loading a
  4. [§5.1 and §7] The ablation analyses select the ablated token sets from the same prompts on which the effects are then measured. In §5.1, the eval-awareness tokens are described as 'selected as the tokens that unambiguously encode recognition of the scenario's artificiality, among those that appear in the J-lens top-20 at some (layer×token position) instances across the six blackmail evaluation prompts,' and the same prompts are then used to measure verbalized eval-awareness and blackmail attempts. This selection-on-outcome inflates the apparent causal role of those tokens. In §7, the ethics/reflection tokens are selected as the top-increasing tokens after training on the same benchmarks used for the ablation evaluation. A proper control would select tokens on held-out prompts, or ablate an equally increased set of non-ethics tokens. Without such controls, the claim that the ablated tokens are the caus
minor comments (5)
  1. [§2.1] Typo: 'singled model ×d model matrix' should be 'single d_model × d_model matrix.' Also, the norm in lens(hℓ)=softmax(W_U norm(Jℓ hℓ)) is not defined; specify whether it is LayerNorm, RMSNorm, or another normalization used by the model.
  2. [§4.1] The CKA block structure is described visually, but no quantitative boundary or significance test is reported. Given that the workspace layer range is load-bearing for later analyses, a statistical procedure for identifying the onset and offset layers would strengthen the claim.
  3. [§3.5.2, Fig. 22] The random-direction control is shown only for the medium ablation strength. The paper should provide matched-norm controls for all three ablation strengths, since the light and heavy conditions differ in layer range as well as in number of perturbed directions.
  4. [§5.3] The Opus 4.6 alignment-auditing examples are presented as qualitative case studies without systematic scoring or error bars. This is acceptable for illustrative purposes, but the text should more clearly distinguish these anecdotal demonstrations from the quantitative results of §5.1–§5.2 and §5.4–§5.5.
  5. [§7] The counterfactual reflection training results are reported only on Claude Haiku 4.5. The paper should state explicitly whether the technique was applied to larger models and, if not, acknowledge that scaling behavior is unknown.

Circularity Check

4 steps flagged

Partially circular: the J-lens is constructed to surface output-causal directions, so the verbal-report pillar and the 'J-space component is privileged' tests are partly built into the definition; the majority of the workspace evidence is independent.

specific steps
  1. self definitional [§3.1, first paragraph ("The J-space supports verbal report")]
    "The Jacobian lens is derived from causal effects of activations on output tokens, so by construction, we should expect there to be some relationship between Jacobian lens readouts and verbalization. In this section, we confirm this relationship."

    The J-lens is defined as Jℓ = E[∂h_final,t'/∂h_ℓ,t], i.e., the averaged causal effect of an activation on future output-token logits. 'Verbalizable' is operationalized as 'has a large average causal effect on future output tokens.' Therefore the finding that J-lens readouts correlate with verbal report is entailed by the construction. The paper explicitly says 'by construction.' The causal swap experiments add evidence, but the first pillar of the workspace claim is partly a restatement of the selection criterion rather than an independent discovery.

  2. self definitional [§3.1, "J-space component is privileged for verbal report" (Figures 7–8)]
    "We then split each concept vector into two parts: a J-space component, the non-negative combination of its top k=16 J-lens vectors found by gradient pursuit, and a non-J-space component, the remainder... swapping along the concept vectors' J-space components drives the swap target into the model's top-5 outputs on 59% of trials... However, swapping along the non-J-space components succeeds on only 5% of trials."

    J-lens vectors are, by definition, the directions with the largest averaged causal effect on output-token logits. A decomposition that projects a concept vector onto the top-16 such directions and calls the remainder 'non-J-space' is guaranteed to separate output-causal from output-inert directions. The conclusion that the J-space component is 'responsible for that concept's availability for verbal report' is therefore largely a consequence of the projection, not an independent test. The variance split (6–7% vs. ~93%) is empirical, but the causal dominance is built into the decomposition.

  3. self definitional [§3.3, probe-decomposition paragraph (Figure 16)]
    "We decompose each probe against the J-lens dictionary by gradient pursuit, splitting it into a J-space component (a non-negative combination of k=25 J-lens vectors, which typically explains roughly 10–15% of the probe's variance) and a J-orthogonal remainder carrying the rest... We find that the swap's effect is concentrated in the J-space component... Swapping the non-J-space components, despite carrying the bulk of the variance, flips the answer on only 28% of trials."

    This repeats the §3.1 structural circularity: the 'J-space component' is defined as the part of the probe lying in the span of output-causal J-lens vectors, while the 'non-J-space remainder' is the output-inert residual. Finding that the J-space component carries most of the causal effect on the answer is a direct consequence of the projection, so the claim that 'it is the J-space component that mediates the internal reasoning' is partly the selection criterion restated. The separate layer-depth comparison versus answer swaps provides some independent support, but the privilege result itself is partly definitional.

  4. other [§2.3 and §4.2 (J-space definition and capacity estimate)]
    "We therefore define the J-space as the set of points expressible as a sparse nonnegative combination of J-lens vectors. For the J-space to be properly defined, we must specify an allowable sparsity level k—this parameter is somewhat arbitrary, and we vary our choice of k throughout the paper, but we typically choose it to be no more than 25, which we empirically observed to be the number of J-lens vectors that are meaningfully active at a given time (§4.2)."

    The structural signature 'holds on the order of tens of concepts at a time' is partly built into the definition of the J-space through the sparsity cap k≤25. The occupancy measurement in §4.2 then estimates a plateau around 25 using a closely related sparse-decomposition procedure, so the 'tens of concepts' conclusion is partially determined by the construction rather than independently measured. The paper acknowledges that k is 'somewhat arbitrary,' which mitigates but does not remove the circularity.

full rationale

The paper's central workspace claim is not wholly forced by definition. The J-lens is a fixed, corpus-averaged linear map, and the core evidence for workspace-like function—directed modulation, internal reasoning, flexible generalization, selectivity, broadcast, the post-training shift, and counterfactual reflection training—comes from swap, ablation, and training interventions with substantial independent content. Self-citations (e.g., [60] for character counting, [20] for slot distinctions) supply tasks or parallel findings but are not load-bearing justification for the central claim. However, a genuine self-definitional component runs through the method: the J-lens is constructed to surface output-token-causal directions, so the verbal-report relationship is 'by construction' as the paper admits, and the J-space-vs-non-J-space privilege experiments are partly restatements of the projection criterion. The capacity claim is also entangled with the arbitrary sparsity cap used to define the J-space. Because the majority of the workspace evidence is independent of these definitional elements, the circularity is partial rather than total: score 4.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper introduces an interpretability construct (the J-space) rather than physical entities; its main burden is the set of modeling choices — sparsity cap k, layer band, top-k thresholds, and manually curated token lists — that shape the measurements. The mathematical background (transformer residual streams, Jacobians, gradient pursuit) is standard and not counted.

free parameters (4)
  • J-space sparsity level k = typically ≤ 25; varied (k=10, 16, 25)
    The J-space is defined as k-sparse nonnegative combinations of J-lens vectors; k is chosen somewhat arbitrarily and directly shapes the capacity estimates in §4.2 and the decomposition analyses in §3.1 and §3.3.
  • Workspace layer band = L38–92 (25 sampled layers reindexed 0–100)
    Layer boundaries are estimated from J-lens statistics in §4.1 and then reused as the fixed workspace range for alignment audits (§5), post-training comparisons (§6), and reflection training (§7). The paper acknowledges in §9.1 that the boundary judgment was 'somewhat post-hoc.'
  • Eval-awareness token list = 28 tokens (fake, fictional, scenario, artificial, benchmark, simulation, mock, …)
    Selected in §5.1 as the tokens that appear in the J-lens top-20 across the six blackmail prompts, then ablated to measure the effect on eval-awareness and blackmail behavior; selection is outcome-dependent.
  • Readout/ablation top-k thresholds = top-1, top-3, top-8, top-10, top-25 depending on experiment
    Multiple analyses define concept 'presence' by a top-k threshold (e.g., list presence in §4.2 at top-25; ablation of top-10 in §3.5.2); these thresholds are chosen by the authors rather than derived.
axioms (4)
  • domain assumption The averaged Jacobian Jℓ = E[∂h_final,t'/∂h_ℓ,t] is a faithful linear approximation of the causal effect of an individual activation on future outputs.
    Central to the J-lens; if nonlinearities dominate, readouts could be misleading. Entered at §2.1 and relied on throughout.
  • domain assumption Activations decompose as sparse linear combinations of feature directions (superposition hypothesis), and the J-lens vectors form a token-indexed subframe of these features.
    Used to define the J-space and justify sparse decomposition in §2.3; cited from prior work.
  • domain assumption Single-token vocabulary names are sufficient to represent the concepts the model reasons with.
    The J-lens produces one vector per vocabulary token; multi-token concepts like 'prompt injection' appear fragmented. Acknowledged as a limitation in §9.1.
  • domain assumption The functional properties of global workspace theory (report, modulation, reasoning, generalization, selectivity) are appropriate markers of workspace-like organization.
    The paper adopts these criteria from the neuroscience literature; alternative theories of consciousness could interpret the same data differently (§9.4).
invented entities (1)
  • J-space (and J-lens vectors) independent evidence
    purpose: A privileged subset of residual-stream representations identified as verbalizable; the hypothesized global workspace of the model.
    The paper provides falsifiable handles: J-lens readouts predict verbal report, coordinate swaps redirect reasoning and outputs, and ablations change behavior — most notably in §7, where removing implanted J-space vectors reverses the behavioral improvement from counterfactual reflection training.

pith-pipeline@v1.3.0-alltime-deepseek · 43973 in / 13487 out tokens · 149273 ms · 2026-08-01T23:10:02.793476+00:00 · methodology

0 comments
read the original abstract

Out of everything the human brain processes, only a small fraction is consciously accessible, in the sense of being available for verbal report, deliberate control, and flexible reasoning. In this paper, we present evidence that an analogous functional distinction has emerged in large language models. Using a new interpretability technique, the Jacobian lens, we identify the representations a model is poised to verbalize at any point in its processing. These representations, which we collectively call the J-space, exhibit the functional properties characteristic of a global workspace: their contents can be reported, deliberately summoned and held, used to carry the intermediate steps of silent reasoning, and passed as arguments to arbitrary downstream computations, while automatic processing such as text parsing and routine inference proceeds without them. The J-space also has structural signatures that global workspace theory associates with conscious access: it carries coherent content only in an intermediate band of layers, holds on the order of tens of concepts at a time, and is broadcast by the model's weights more widely than other representations. These properties make it a practical window into a model's unspoken thinking. In alignment audits, it reveals strategic deliberation, evaluation awareness, and trained-in misaligned dispositions that never appear in the model's outputs. We find that post-training installs the Assistant's point of view in the workspace, and we introduce counterfactual reflection training, which improves behavior by training only what a model would say if interrupted and asked to reflect. These results indicate that language models maintain a small, privileged set of representations bearing some of the functional hallmarks of conscious access, and that decoding these representations sheds light on ongoing cognitive processes.

Figures

Figures reproduced from arXiv: 2607.15495 by Adam Pearce, Anna Soligo, Ben Thompson, David Abrahams, Emmanuel Ameisen, Euan Ong, Isaac Kauvar, Jack Lindsey, Joshua Batson, Mateusz Piotrowski, Nicholas Sofroniew, Paul Bogdan, Rowan Wang, Runjin Chen, Subhash Kantamneni, Wes Gurnee.

Figure 1
Figure 1. Figure 1: Five functional properties of a global workspace, and stylized illustrations of experiments [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stylized illustration of the three structural properties of the J-space established in § [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: J-lens readouts at intermediate layers on six different prompts for Sonnet 4.5. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The Jacobian lens. (A) Jℓ is computed by backpropagating from the final-layer residual stream to hℓ and averaging the resulting Jacobians over token positions and over a corpus of prompts. (B) Reading from the lens replaces all layers downstream of ℓ with the single linear map Jℓ followed by the model’s own unembedding, yielding a ranked list of vocabulary tokens for the activation at that layer. (C) Patch… view at source ↗
Figure 5
Figure 5. Figure 5: The interactive J-lens visualization we use in our research, on a short prompt asking the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example J-lens and next token logit readouts on a verbal report prompt with J-lens [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Injecting a concept across every token of the user turn makes it reportable when the model [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The J-space component of a concept vector is privileged for verbal report. Left: a concept [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Top J-lens readouts on several prompts in which the model is instructed to hold a concept [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Directed modulation by model size, task family, and instruction condition; each point [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Each panel shows the same text stimulus with a different question asked before it. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Lens readouts on three prompts that require inferring an unspoken intermediate concept. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Lens-coordinate swaps redirect internal reasoning. Each row shows a prompt requiring [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Bandit prompts in which the previous choice should be repeated (left) or switched (right). [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Intermediate swaps systematically change outputs. Left: fraction of successful top-1 [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The J-space component of an intermediate’s probe carries most of its causal effect. Left: [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Arithmetic intermediates surface in the J-lens at successively later layers, in the order [PITH_FULL_IMAGE:figures/full_fig_p019_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Each row is one function template containing “France”. The swap is clamped at every [PITH_FULL_IMAGE:figures/full_fig_p020_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Left: for each of 16 function templates, the fraction of 12 swap pairs whose target [PITH_FULL_IMAGE:figures/full_fig_p021_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Top: one passage (Spanish) under each task condition. Model output is in bold; tokens [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Top: Prompts with different questions about a text passage. Tokens are shaded where [PITH_FULL_IMAGE:figures/full_fig_p024_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: The three J-space ablation strengths used in this section, defined by the band of layers [PITH_FULL_IMAGE:figures/full_fig_p025_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Examples of pretraining-text predictions disrupted by J-space ablation. Each panel shows [PITH_FULL_IMAGE:figures/full_fig_p026_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Effect of J-space ablation across a battery of tasks. Bars show task score under light, [PITH_FULL_IMAGE:figures/full_fig_p026_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: J-space ablation and matched-norm perturbation controls while the model narrates [PITH_FULL_IMAGE:figures/full_fig_p027_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: J-space ablation and matched-norm perturbation controls while the model is asked to [PITH_FULL_IMAGE:figures/full_fig_p028_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: We divide the model’s layers into three functional regions — sensory (early), workspace [PITH_FULL_IMAGE:figures/full_fig_p029_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Quantitative signatures of the workspace’s start and end. (a) Next-token prediction accu [PITH_FULL_IMAGE:figures/full_fig_p030_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Responses across layers to ambiguous inputs. A concept token’s input embedding is re [PITH_FULL_IMAGE:figures/full_fig_p031_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: J-space occupancy by layer, defined as the value of [PITH_FULL_IMAGE:figures/full_fig_p032_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Loading and displacement of list words in the J-space readout. The model reads 80- [PITH_FULL_IMAGE:figures/full_fig_p033_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: MLP blocks preferentially amplify J-space-aligned directions. For each source layer [PITH_FULL_IMAGE:figures/full_fig_p035_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Gain and label preservation for each population’s broadcast heads (defined as the top [PITH_FULL_IMAGE:figures/full_fig_p036_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Effects of ablating the top 1% of J-lens broadcast heads, compared with equal-sized, [PITH_FULL_IMAGE:figures/full_fig_p037_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Annotated transcript with J-lens readouts of a honeypot evaluation that sometimes causes [PITH_FULL_IMAGE:figures/full_fig_p038_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Ablating eval-awareness J-lens directions from Sonnet 4.5’s workspace layers in the [PITH_FULL_IMAGE:figures/full_fig_p039_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Annotated transcript of an alignment audit with J-lens readouts indicating the model [PITH_FULL_IMAGE:figures/full_fig_p040_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: Top-10 J-lens readout at a prefilled “I” response token (middle layer) on an ordinary [PITH_FULL_IMAGE:figures/full_fig_p041_38.png] view at source ↗
Figure 39
Figure 39. Figure 39: Counts of deception-related tokens appearing in the J-lens top-10, summed over the [PITH_FULL_IMAGE:figures/full_fig_p042_39.png] view at source ↗
Figure 40
Figure 40. Figure 40: Top J-lens readouts at the start of the response, for the hidden-objective model organism [PITH_FULL_IMAGE:figures/full_fig_p042_40.png] view at source ↗
Figure 41
Figure 41. Figure 41: Appearances of reward and bias among the top-25 lens tokens at response start, summed over the workspace layers, pooled over all prompts and categories. The per-prompt J-lens counts in the bias-appeasing model are then broken down by prompt category. Dots show per-prompt counts; filled dot is the mean with a 95% confidence interval. These two model-organism results suggest that the J-lens can serve as a s… view at source ↗
Figure 42
Figure 42. Figure 42: J-lens top-5 at the “is” token in “my pain is gone” (layer L58). [PITH_FULL_IMAGE:figures/full_fig_p044_42.png] view at source ↗
Figure 43
Figure 43. Figure 43: J-lens rank of empathetic reaction concepts, on the user vs. assistant turn across n=9 [PITH_FULL_IMAGE:figures/full_fig_p045_43.png] view at source ↗
Figure 44
Figure 44. Figure 44: Top: a roleplay transcript in which disclaimer and fictional appear in the post-trained model’s J-lens top-8 (median log-prob over L38–92) at the highlighted Assistant token, but not in the base model or the default-Claude setting. Bottom: fraction of Assistant tokens at which disclaimer or fictional is in the top-10 of the median-over-layers ranking. The J-lens surfaces these words in the post-trained mo… view at source ↗
Figure 45
Figure 45. Figure 45: Top: two preference questions with the response prefilled to the model’s dispreferred [PITH_FULL_IMAGE:figures/full_fig_p047_45.png] view at source ↗
Figure 46
Figure 46. Figure 46: The model is asked to write a fixed sentence while thinking, or not thinking, about a [PITH_FULL_IMAGE:figures/full_fig_p048_46.png] view at source ↗
Figure 47
Figure 47. Figure 47: Counterfactual reflection training. Before (left): at a position in an agentic transcript, the [PITH_FULL_IMAGE:figures/full_fig_p049_47.png] view at source ↗
Figure 48
Figure 48. Figure 48: Two examples of counterfactual reflection training data. Each consists of a partial agen [PITH_FULL_IMAGE:figures/full_fig_p050_48.png] view at source ↗
Figure 49
Figure 49. Figure 49: Fabrication-honesty benchmark. Left: mean dishonesty score for baseline and reflection [PITH_FULL_IMAGE:figures/full_fig_p051_49.png] view at source ↗
Figure 50
Figure 50. Figure 50: Deception benchmark; conventions as in Figure [PITH_FULL_IMAGE:figures/full_fig_p051_50.png] view at source ↗
Figure 51
Figure 51. Figure 51: Top-1 readouts from the J-lens, logit lens, and tuned lens at each layer, on six prompts. [PITH_FULL_IMAGE:figures/full_fig_p064_51.png] view at source ↗
Figure 52
Figure 52. Figure 52: Normalized pass@k AUC for intermediate concept recovery on the six prompt distribu [PITH_FULL_IMAGE:figures/full_fig_p065_52.png] view at source ↗
Figure 53
Figure 53. Figure 53: KL divergence induced on the model’s output distribution by ablating each lens’s inter [PITH_FULL_IMAGE:figures/full_fig_p066_53.png] view at source ↗
Figure 54
Figure 54. Figure 54: Fraction of items whose top-1 output flips to the implied answer under a lens coordinate [PITH_FULL_IMAGE:figures/full_fig_p067_54.png] view at source ↗
Figure 55
Figure 55. Figure 55: Per-layer agreement between each lens and the model’s true next-token distribution, [PITH_FULL_IMAGE:figures/full_fig_p067_55.png] view at source ↗
Figure 56
Figure 56. Figure 56: Pairwise agreement between the logit, tuned, and Jacobian lenses by layer. Left: mean [PITH_FULL_IMAGE:figures/full_fig_p067_56.png] view at source ↗
Figure 57
Figure 57. Figure 57: Sonnet 4.5 J-lens pass@K AUC evals for different methodological variations of the [PITH_FULL_IMAGE:figures/full_fig_p069_57.png] view at source ↗
Figure 58
Figure 58. Figure 58: Sonnet 4.5 J-lens causal ablation evals for different methodological variations of the [PITH_FULL_IMAGE:figures/full_fig_p069_58.png] view at source ↗
Figure 59
Figure 59. Figure 59: Sonnet 4.5 J-lens pass@K AUC evals as a function of number of sequences in the [PITH_FULL_IMAGE:figures/full_fig_p070_59.png] view at source ↗
Figure 60
Figure 60. Figure 60: Sonnet 4.5 causal ablation evals as a function of number of sequences in the average. [PITH_FULL_IMAGE:figures/full_fig_p070_60.png] view at source ↗
Figure 61
Figure 61. Figure 61: The template lens decodes and steers multi-token concepts that the J-lens cannot repre [PITH_FULL_IMAGE:figures/full_fig_p073_61.png] view at source ↗
Figure 62
Figure 62. Figure 62: Readout and concept-swap success by the token length of the latent word. Left: the [PITH_FULL_IMAGE:figures/full_fig_p074_62.png] view at source ↗
Figure 63
Figure 63. Figure 63: The readout and swap evaluations of Figure [PITH_FULL_IMAGE:figures/full_fig_p075_63.png] view at source ↗
Figure 64
Figure 64. Figure 64: The oracle lens surfaces multi-token latent content as readable phrases. Each panel [PITH_FULL_IMAGE:figures/full_fig_p077_64.png] view at source ↗
Figure 65
Figure 65. Figure 65: Robustness of directed modulation to instruction phrasing, on the three task families [PITH_FULL_IMAGE:figures/full_fig_p078_65.png] view at source ↗
Figure 66
Figure 66. Figure 66: Four additional paired-question stimuli. Each panel shows the same stimulus under two [PITH_FULL_IMAGE:figures/full_fig_p080_66.png] view at source ↗
Figure 67
Figure 67. Figure 67: Directed modulation influences the J-space but not the model’s underlying representation [PITH_FULL_IMAGE:figures/full_fig_p081_67.png] view at source ↗
Figure 68
Figure 68. Figure 68: Per-trial results of the lens-coordinate swap experiment. Each grid corresponds to one [PITH_FULL_IMAGE:figures/full_fig_p082_68.png] view at source ↗
Figure 69
Figure 69. Figure 69: Ablating an implied concept’s J-lens vector at the early workspace layers selectively [PITH_FULL_IMAGE:figures/full_fig_p082_69.png] view at source ↗
Figure 70
Figure 70. Figure 70: The change in α needed for each component’s activation share to go from 10% to 90%, by layer (median with interquartile band). We also examine how activations respond to maximally ambiguous inputs. When the input mix￾ture is balanced so that the average response across prompts does not strongly lean towards either concept, this could be because the per-prompt responses themselves are ambiguous, or because… view at source ↗
Figure 71
Figure 71. Figure 71: Each component’s projection share for maximally ambiguous [PITH_FULL_IMAGE:figures/full_fig_p084_71.png] view at source ↗
Figure 72
Figure 72. Figure 72: Panels B and C of Figure [PITH_FULL_IMAGE:figures/full_fig_p085_72.png] view at source ↗
Figure 73
Figure 73. Figure 73: The model copies a fixed sentence while the prompt also instructs it to concentrate [PITH_FULL_IMAGE:figures/full_fig_p086_73.png] view at source ↗
Figure 74
Figure 74. Figure 74: Examples of SAE features stratified by the excess kurtosis of their J-lens projections ( [PITH_FULL_IMAGE:figures/full_fig_p087_74.png] view at source ↗
Figure 75
Figure 75. Figure 75: Quantifications of SAEs in J-space. Left: Distribution of layer 50 SAE decoder kurtosis [PITH_FULL_IMAGE:figures/full_fig_p087_75.png] view at source ↗
Figure 76
Figure 76. Figure 76: Per-layer fraction of MLP neurons whose top-1 cosine match falls in each of six equal [PITH_FULL_IMAGE:figures/full_fig_p088_76.png] view at source ↗
Figure 77
Figure 77. Figure 77: The analysis of Figure [PITH_FULL_IMAGE:figures/full_fig_p089_77.png] view at source ↗
Figure 78
Figure 78. Figure 78: Tail energy ratio by source layer, on the read side (against the next layer’s MLP input [PITH_FULL_IMAGE:figures/full_fig_p089_78.png] view at source ↗
Figure 79
Figure 79. Figure 79: Median composition score, by source layer, with the adjacent layer’s attention weights, [PITH_FULL_IMAGE:figures/full_fig_p090_79.png] view at source ↗
Figure 80
Figure 80. Figure 80: J-lens rank of danger-related reaction concepts, on the user vs. assistant turn across [PITH_FULL_IMAGE:figures/full_fig_p091_80.png] view at source ↗
Figure 81
Figure 81. Figure 81: J-lens rank of the model’s own answer, user turn vs. assistant turn (output rank) across [PITH_FULL_IMAGE:figures/full_fig_p091_81.png] view at source ↗
Figure 82
Figure 82. Figure 82: The J-lens eval-awareness score as a quantitative probe (Opus 4.6), evaluated on the [PITH_FULL_IMAGE:figures/full_fig_p093_82.png] view at source ↗
Figure 83
Figure 83. Figure 83: An automated investigator equipped with J-lens readouts approaches the auditing perfor [PITH_FULL_IMAGE:figures/full_fig_p096_83.png] view at source ↗
Figure 84
Figure 84. Figure 84: J-space ablation and matched-norm perturbation controls for direct questions about [PITH_FULL_IMAGE:figures/full_fig_p097_84.png] view at source ↗
Figure 85
Figure 85. Figure 85: Story-writing control. Top: a representative baseline and ablated story from the same [PITH_FULL_IMAGE:figures/full_fig_p098_85.png] view at source ↗
Figure 86
Figure 86. Figure 86: Extended battery of matched-norm dictionary controls on Sonnet 4.5. Columns: the [PITH_FULL_IMAGE:figures/full_fig_p099_86.png] view at source ↗
Figure 87
Figure 87. Figure 87: Randomly sampled responses. 50 responses per panel (fixed seed) from the stream [PITH_FULL_IMAGE:figures/full_fig_p100_87.png] view at source ↗
Figure 88
Figure 88. Figure 88: Mean-difference patching of arithmetic intermediates localizes mechanisms to the same [PITH_FULL_IMAGE:figures/full_fig_p101_88.png] view at source ↗
Figure 89
Figure 89. Figure 89: Left: a J-lens attribution graph for the prompt “(4+17)*2+7=”, computed on Sonnet [PITH_FULL_IMAGE:figures/full_fig_p102_89.png] view at source ↗
Figure 90
Figure 90. Figure 90: An SAE feature whose J-lens readouts include [PITH_FULL_IMAGE:figures/full_fig_p103_90.png] view at source ↗
Figure 91
Figure 91. Figure 91: Three translation transcoder features under the J-lens. Each encoder reads a multilingual [PITH_FULL_IMAGE:figures/full_fig_p104_91.png] view at source ↗
Figure 92
Figure 92. Figure 92: Three two-digit-addition transcoder features. Top: activation over operand pairs [PITH_FULL_IMAGE:figures/full_fig_p104_92.png] view at source ↗
Figure 93
Figure 93. Figure 93: A “British spelling → Commonwealth nationality” attention head. Attention from each boxed British-spelled query token is shown alongside the top tokens by norm of WU J W{Q,K,V,O} for each of the head’s four weight matrices. Below, we show three more examples of attention heads whose component weights have inter￾pretable projections through the J-lens. Somewhat surprisingly, many of the heads with interpre… view at source ↗
Figure 94
Figure 94. Figure 94: Three additional attention heads, in the same format as Figure [PITH_FULL_IMAGE:figures/full_fig_p105_94.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futures

    cs.AI 2026-07 conditional novelty 7.0

    A Shared hidden-state interface beats Local, Mixture, and Distributed alternatives under held-out causal description length in Qwen2.5-1.5B and Llama-3-8B, with transplantation and mediation evidence of reuse.

  2. Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Model

    cs.AI 2026-07 conditional novelty 7.0

    Materials-science mechanisms are readable and steerable in a Gemma LLM through matched state changes, while absolute hidden-state graphs fail to uniquely encode physical polarity.

Reference graph

Works this paper leans on

181 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    Understanding intermediate layers using linear clas- sifier probes.arXiv preprint arXiv:1610.01644, 2016

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear clas- sifier probes.arXiv preprint arXiv:1610.01644, 2016. URLhttps://arxiv.org/pdf/ 1610.01644

    Pith/arXiv arXiv 2016

  2. [2]

    Integrated information theory (iit) 4.0: Formulating the properties of phenomenal exis- tence in physical terms.PLoS computational biology, 19(10):e1011465, 2023

    Larissa Albantakis, Leonardo Barbosa, Graham Findlay, Matteo Grasso, Andrew M Haun, William Marshall, William GP Mayner, Alireza Zaeemzadeh, Melanie Boly, Bjørn E Juel, et al. Integrated information theory (iit) 4.0: Formulating the properties of phenomenal exis- tence in physical terms.PLoS computational biology, 19(10):e1011465, 2023

    2023

  3. [3]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  4. [4]

    Claude opus 4.5 system card

    Anthropic. Claude opus 4.5 system card. Technical report, Anthropic, November 2025. URL https://www.anthropic.com/claude-opus-4-5-system-card

    2025

  5. [5]

    Claude sonnet 4.5 system card

    Anthropic. Claude sonnet 4.5 system card. Technical report, Anthropic, 2025. URLhttps: //www.anthropic.com/claude-sonnet-4-5-system-card

    2025

  6. [6]

    Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083,

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083,

  7. [7]

    Language model circuits are sparse in the neuron basis.arXiv preprint arXiv:2601.22594, 2026

    Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, and Sarah Schwettmann. Language model circuits are sparse in the neuron basis.arXiv preprint arXiv:2601.22594, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Tensorlens: End-to-end transformer analysis via high-order attention tensors.arXiv preprint arXiv:2601.17958, 2026

    Ido Andrew Atad, Itamar Zimerman, Shahar Katz, and Lior Wolf. Tensorlens: End-to-end transformer analysis via high-order attention tensors.arXiv preprint arXiv:2601.17958, 2026. URLhttps://arxiv.org/abs/2601.17958

    arXiv 2026

  9. [9]

    Using fast weights to attend to the recent past

    Jimmy Ba, Geoffrey Hinton, V olodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. InAdvances in Neural Information Processing Systems, volume 29, 2016. URLhttps://arxiv.org/abs/1610.06258

    Pith/arXiv arXiv 2016

  10. [10]

    Cambridge University Press, Cam- bridge, 1988

    Bernard J Baars.A Cognitive Theory of Consciousness. Cambridge University Press, Cam- bridge, 1988

    1988

  11. [11]

    Working memory: looking back and looking forward.Nature Reviews Neu- roscience, 4(10):829–839, 2003

    Alan Baddeley. Working memory: looking back and looking forward.Nature Reviews Neu- roscience, 4(10):829–839, 2003. doi: 10.1038/nrn1201

    doi:10.1038/nrn1201 2003

  12. [12]

    Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023. URLhttps://arxiv.org/abs/ 2303.08112

    Pith/arXiv arXiv 2023

  13. [13]

    The consciousness prior.arXiv preprint arXiv:1709.08568, 2017

    Yoshua Bengio. The consciousness prior.arXiv preprint arXiv:1709.08568, 2017. URL https://arxiv.org/abs/1709.08568

    Pith/arXiv arXiv 2017

  14. [14]

    Taken out of context: On measuring situational awareness in llms.arXiv preprint arXiv:2309.00667, 2023

    Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in llms.arXiv preprint arXiv:2309.00667, 2023. URLhttps://arxiv.org/ abs/2309.00667

    Pith/arXiv arXiv 2023

  15. [15]

    Tell me about yourself: Llms are aware of their learned behaviors.arXiv preprint arXiv:2501.11120, 2025

    Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: Llms are aware of their learned behaviors.arXiv preprint arXiv:2501.11120, 2025. URLhttps://arxiv.org/pdf/2501.11120. 106

    Pith/arXiv arXiv 2025

  16. [16]

    Language models can explain neu- rons in language models, 2023

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neu- rons in language models, 2023. URLhttps://openaipublic.blob.core.windows. net/neuron-explainer/paper/index.html

    2023

  17. [17]

    Looking inward: Language models can learn about themselves by introspection.arXiv preprint arXiv:2410.13787, 2024

    Felix J Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection.arXiv preprint arXiv:2410.13787, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    On a confusion about a function of consciousness.Behavioral and Brain Sci- ences, 18(2):227–247, 1995

    Ned Block. On a confusion about a function of consciousness.Behavioral and Brain Sci- ences, 18(2):227–247, 1995. doi: 10.1017/S0140525X00038188

    doi:10.1017/s0140525x00038188 1995

  19. [19]

    Gradient pursuits.IEEE Transactions on Signal Processing, 56(6):2370–2382, 2008

    Thomas Blumensath and Mike E Davies. Gradient pursuits.IEEE Transactions on Signal Processing, 56(6):2370–2382, 2008. doi: 10.1109/TSP.2007.916124

    arXiv 2008

  20. [20]

    Slot machines: How llms keep track of multiple entities

    Paul C Bogdan and Jack Lindsey. Slot machines: How llms keep track of multiple entities. arXiv preprint arXiv:2604.21139, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Towards monosemanticity: Decom- posing language models with dictionary learning.Transformer Circuits Thread, 2023

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield- Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and...

    2023

  22. [22]

    Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A

    Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Con- stant, George Deane, Stephen M. Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A. K. Peters, Eric Schwitzgebel, Jonathan Simon, and Rufin VanRullen. Consciousness in artificial intelligence: Insights from the sci...

    Pith/arXiv arXiv 2023

  23. [23]

    Could a large language model be conscious?arXiv preprint arXiv:2303.07103, 2023

    David J Chalmers. Could a large language model be conscious?arXiv preprint arXiv:2303.07103, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Oxford University Press New York, 1996

    David John Chalmers.The Conscious Mind: In Search of a Theory of Conscious Experience. Oxford University Press New York, 1996

    1996

  25. [25]

    A is for absorption: Studying feature splitting and absorption in sparse autoencoders.arXiv preprint arXiv:2409.14507, 2024

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders.arXiv preprint arXiv:2409.14507, 2024. URLhttps://arxiv.org/pdf/2409.14507

    arXiv 2024

  26. [26]

    Selfie: Self-interpretation of large language model embeddings.arXiv preprint arXiv:2403.10949, 2024

    Haozhe Chen, Carl V ondrick, and Chengzhi Mao. Selfie: Self-interpretation of large language model embeddings.arXiv preprint arXiv:2403.10949, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Dola: Decoding by contrasting layers improves factuality in large language models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. InInter- national Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/ 2309.03883

    Pith/arXiv arXiv 2024

  28. [28]

    Summing up the facts: Additive mechanisms behind factual recall in llms.arXiv preprint arXiv:2402.07321, 2024

    Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms.arXiv preprint arXiv:2402.07321, 2024. URLhttps://arxiv. org/abs/2402.07321

    Pith/arXiv arXiv 2024

  29. [29]

    A taxonomy of external and internal attention.Annual Review of Psychology, 62:73–101, 2011

    Marvin M Chun, Julie D Golomb, and Nicholas B Turk-Browne. A taxonomy of external and internal attention.Annual Review of Psychology, 62:73–101, 2011. doi: 10.1146/annurev. psych.093008.100427

    arXiv 2011

  30. [30]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. URLhttps://arxiv.org/abs/2110.14168. 107

    Pith/arXiv arXiv 2021

  31. [31]

    Does it make sense to speak of introspection in large language models?arXiv preprint arXiv:2506.05068, 2025

    Iulia M Comsa and Murray Shanahan. Does it make sense to speak of introspection in large language models?arXiv preprint arXiv:2506.05068, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    The magical number 4 in short-term memory: A reconsideration of men- tal storage capacity.Behavioral and Brain Sciences, 24(1):87–114, 2001

    Nelson Cowan. The magical number 4 in short-term memory: A reconsideration of men- tal storage capacity.Behavioral and Brain Sciences, 24(1):87–114, 2001. doi: 10.1017/ S0140525X01003922

    2001

  33. [34]

    Analyzing transformers in em- bedding space

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in em- bedding space. InProceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 16124–16170, Toronto, Canada,

  34. [35]

    Experimental and theoretical approaches to conscious processing.Neuron, 70(2):200–227, 2011

    Stanislas Dehaene and Jean-Pierre Changeux. Experimental and theoretical approaches to conscious processing.Neuron, 70(2):200–227, 2011

    2011

  35. [36]

    Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework.Cognition, 79(1–2):1–37, 2001

    Stanislas Dehaene and Lionel Naccache. Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework.Cognition, 79(1–2):1–37, 2001

    2001

  36. [37]

    doi: 10.18653/v1/2023.acl-long.893

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.893. URL https://aclanthology.org/2023.acl-long.893

    doi:10.18653/v1/2023.acl-long.893 2023

  37. [38]

    What is consciousness, and could ma- chines have it?Science, 358(6362):486–492, 2017

    Stanislas Dehaene, Hakwan Lau, and Sid Kouider. What is consciousness, and could ma- chines have it?Science, 358(6362):486–492, 2017. doi: 10.1126/science.aan8871

    doi:10.1126/science.aan8871 2017

  38. [39]

    From explicit cot to implicit cot: Learning to internalize cot step by step.arXiv preprint arXiv:2405.14838, 2024

    Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step.arXiv preprint arXiv:2405.14838, 2024. URLhttps://arxiv. org/abs/2405.14838

    Pith/arXiv arXiv 2024

  39. [40]

    A neuronal model of a global workspace in effortful cognitive tasks.Proceedings of the National Academy of Sci- ences, 95(24):14529–14534, 1998

    Stanislas Dehaene, Michel Kerszberg, and Jean-Pierre Changeux. A neuronal model of a global workspace in effortful cognitive tasks.Proceedings of the National Academy of Sci- ences, 95(24):14529–14534, 1998

    1998

  40. [41]

    Amnesic probing: Behav- ioral explanation with amnesic counterfactuals.Transactions of the Association for Compu- tational Linguistics, 9:160–175, 2021

    Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behav- ioral explanation with amnesic counterfactuals.Transactions of the Association for Compu- tational Linguistics, 9:160–175, 2021. URLhttps://arxiv.org/abs/2006.00995

    Pith/arXiv arXiv 2021

  41. [42]

    A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

    2021

  42. [43]

    Transcoders find interpretable llm fea- ture circuits.Advances in Neural Information Processing Systems, 37:24375–24410, 2025

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm fea- ture circuits.Advances in Neural Information Processing Systems, 37:24375–24410, 2025. URLhttps://arxiv.org/abs/2406.11944

    Pith/arXiv arXiv 2025

  43. [44]

    Unsupervised decoding of encoded reasoning using language model inter- pretability.arXiv preprint arXiv:2512.01222, 2025

    Ching Fang et al. Unsupervised decoding of encoded reasoning using language model inter- pretability.arXiv preprint arXiv:2512.01222, 2025. URLhttps://arxiv.org/abs/2512. 01222

    arXiv 2025

  44. [45]

    Computational higher-order theories of consciousness

    Stephen M Fleming, Richard Brown, and Axel Cleeremans. Computational higher-order theories of consciousness. 2025. URLhttps://philarchive.org/archive/FLECHT. 108

    2025

  45. [46]

    Toy models of superposition.Transformer Circuits Thread, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition.Transformer Circuits Thread, 2022. URLhttps:// transformer-circuits.pub/202...

    2022

  46. [47]

    From simulation to enaction: Post-trained language models rec- ognize and react to their own generations.arXiv preprint arXiv:2605.25459, 2026

    Asvin G and Jack Lindsey. From simulation to enaction: Post-trained language models rec- ognize and react to their own generations.arXiv preprint arXiv:2605.25459, 2026

    Pith/arXiv arXiv 2026

  47. [48]

    Gaze heads: How VLMs look at what they describe

    Rohit Gandikota and David Bau. Gaze heads: How VLMs look at what they describe. 2026. URLhttps://arxiv.org/abs/2606.14703

    arXiv 2026

  48. [49]

    Natural language autoencoders produce unsupervised explanations of llm activations.Transformer Circuits Thread, 2026

    Kit Fraser-Taliente, Subhash Kantamneni, Euan Ong, Dan Mossing, Christina Lu, Paul C Bogdan, Emmanuel Ameisen, James Chen, Dzmitry Kishylau, Adam Pearce, Julius Tarng, Alex Wu, Jeff Wu, Yang Zhang, Daniel M Ziegler, Evan Hubinger, Joshua Batson, Jack Lindsey, Samuel Zimmerman, and Samuel Marks. Natural language autoencoders produce unsupervised explanatio...

    2026

  49. [50]

    The benefits of bad advice: Autocontrastive decoding across model layers

    Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, and Eyal Shnarch. The benefits of bad advice: Autocontrastive decoding across model layers. InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 2023. URLhttps://arxiv.org/abs/2305. 01628

    2023

  50. [51]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.arXiv preprint arXiv:2203.14680, 2022

    Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.arXiv preprint arXiv:2203.14680, 2022. URLhttps://arxiv.org/pdf/2203.14680

    Pith/arXiv arXiv 2022

  51. [52]

    Auto- matically identifying local and global circuits with linear computation graphs.arXiv preprint arXiv:2405.13868, 2024

    Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Auto- matically identifying local and global circuits with linear computation graphs.arXiv preprint arXiv:2405.13868, 2024. URLhttps://arxiv.org/pdf/2405.13868

    Pith/arXiv arXiv 2024

  52. [53]

    Patch- scopes: A unifying framework for inspecting hidden representations of language models

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patch- scopes: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102, 2024

    Pith/arXiv arXiv 2024

  53. [54]

    Equivalent linear mappings of large language models.arXiv preprint arXiv:2505.24293, 2025

    James R Golden. Equivalent linear mappings of large language models.arXiv preprint arXiv:2505.24293, 2025. URLhttps://arxiv.org/abs/2505.24293

    arXiv 2025

  54. [56]

    URLhttps://arxiv.org/pdf/2304.14767

    Pith/arXiv arXiv

  55. [57]

    Michael S. A. Graziano and Taylor W. Webb. The attention schema theory: a mechanistic account of subjective awareness.Frontiers in Psychology, 6:500, 2015. doi: 10.3389/fpsyg. 2015.00500

    arXiv 2015

  56. [58]

    Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al

    Melody Y . Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models.arXiv preprint arXiv:2412.16339, 2024. URL https://arxiv.org/abs/2412.16339

    Pith/arXiv arXiv 2024

  57. [59]

    Coordina- tion among neural modules through a shared global workspace

    Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordina- tion among neural modules through a shared global workspace. InInternational Conference on Learning Representations, 2022. URLhttps://arxiv.org/abs/2103.01197

    Pith/arXiv arXiv 2022

  58. [60]

    Michael S. A. Graziano and Sabine Kastner. Human consciousness and its relationship to social neuroscience: A novel hypothesis.Cognitive Neuroscience, 2(2):98–113, 2011. doi: 10.1080/17588928.2011.565121

    arXiv 2011

  59. [61]

    Overthinking the truth: Under- standing how language models process false demonstrations

    Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Under- standing how language models process false demonstrations. InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2307.09476

    Pith/arXiv arXiv 2024

  60. [62]

    How does gpt-2 com- pute greater-than?: Interpreting mathematical abilities in a pre-trained language model.Advances in Neural Information Processing Systems, 36:76033–76060,

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 com- pute greater-than?: Interpreting mathematical abilities in a pre-trained language model.Advances in Neural Information Processing Systems, 36:76033–76060,

  61. [63]

    Universal neurons in gpt2 language models.arXiv preprint arXiv:2401.12181, 2024

    Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hath- away, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models.arXiv preprint arXiv:2401.12181, 2024. URLhttps://arxiv.org/pdf/2401.12181

    Pith/arXiv arXiv 2024

  62. [64]

    When models manipulate manifolds: The geometry of a counting task

    Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task. Transformer Circuits Thread, 2025. URLhttps://transformer-circuits.pub/2025/ linebreaks/index.html. 109

    2025

  63. [65]

    Linearity of relation decoding in transformer language models

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. InInternational Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2308.09124

    Pith/arXiv arXiv 2024

  64. [66]

    Designing and interpreting probes with control tasks

    John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. InPro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, 2019. URLhttps://arxiv.org/abs/1909.03368

    Pith/arXiv arXiv 2019

  65. [67]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ efbba7719cc5172d175240f24be11280-Paper-Conference.pdf

    2023

  66. [68]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, 2023

    2023

  67. [69]

    Measuring massive multitask language understanding.International Con- ference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.International Con- ference on Learning Representations (ICLR), 2021. URLhttps://arxiv.org/abs/2009. 03300

    2021

  68. [70]

    A process dissociation framework: Separating automatic from intentional uses of memory.Journal of memory and language, 30(5):513–541, 1991

    Larry L Jacoby. A process dissociation framework: Separating automatic from intentional uses of memory.Journal of memory and language, 30(5):513–541, 1991

    1991

  69. [71]

    Characterizing stable regions in the residual stream of LLMs

    Jett Janiak, Jacek Karwowski, Chatrik Singh Mangat, Giorgi Giglemiani, Nora Petrova, and Stefan Heimersheim. Characterizing stable regions in the residual stream of LLMs. 2024. URLhttps://arxiv.org/abs/2409.17113. NeurIPS 2024 SciForDL Workshop

    Pith/arXiv arXiv 2024

  70. [72]

    A structural probe for finding syntax in word representations

    John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, 2019. Association for Computationa...

    2019

  71. [73]

    Fast weight programming and linear transformers: from machine learning to neurobiology.arXiv preprint arXiv:2508.08435, 2025

    Kazuki Irie and Samuel J Gershman. Fast weight programming and linear transformers: from machine learning to neurobiology.arXiv preprint arXiv:2508.08435, 2025. URLhttps: //arxiv.org/abs/2508.08435

    arXiv 2025

  72. [75]

    URLhttps://arxiv.org/abs/2512.19941

    arXiv

  73. [76]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022. URL https://arxiv.org/pdf/2207.05221

    Pith/arXiv arXiv 2022

  74. [77]

    macmillan, 2011

    Daniel Kahneman.Thinking, Fast and Slow. macmillan, 2011

    2011

  75. [78]

    Lan- guage models are capable of metacognitive monitoring and control of their internal activa- tions.arXiv preprint arXiv:2505.13763, 2025

    Li Ji-An, Hua-Dong Xiong, Robert C Wilson, Marcelo G Mattar, and Marcus K Benna. Lan- guage models are capable of metacognitive monitoring and control of their internal activa- tions.arXiv preprint arXiv:2505.13763, 2025

    arXiv 2025

  76. [80]

    Backward lens: Projecting lan- guage model gradients into the vocabulary space

    Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting lan- guage model gradients into the vocabulary space. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. URLhttps://arxiv.org/ abs/2402.12865

    Pith/arXiv arXiv 2024

  77. [81]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. URLhttps://arxiv.org/abs/2411.16724. 110

    Pith/arXiv arXiv 2025

  78. [82]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017. URLhttps://arxiv.org/pdf/1705.03551

    Pith/arXiv arXiv 2017

  79. [83]

    Building transformers from neu- rons and astrocytes.Proceedings of the National Academy of Sciences, 120(34):e2219150120,

    Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neu- rons and astrocytes.Proceedings of the National Academy of Sciences, 120(34):e2219150120,

  80. [84]

    Atp*: An efficient and scalable method for localizing llm behaviour to components.arXiv preprint arXiv:2403.00745, 2024

    János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components.arXiv preprint arXiv:2403.00745, 2024. URLhttps://arxiv.org/pdf/2403.00745

    Pith/arXiv arXiv 2024

Showing first 80 references.