Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transformer's hidden states sit nearly perpendicular to its answer on purpose, and that off-axis frame can be prescribed before training.

desk verdict Off-axis insulation is a real, well-measured result; fix the logit-lens inconsistency and release code before betting too hard on the frame-prescription story. read the letter →

arxiv 2608.10251 v1 pith:XFJSBDYJ submitted 2026-08-10 cs.CL cs.LG

Mark Oskin This is my paper · ORCID
classification cs.CLcs.LG
keywords off-axisrepresentationresidualstreamlogitlensattentionmixingparticipationratiogaugefreedomearlyexitpressuretwo-phasecomputation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the off-axis position of a transformer's intermediate states—usually treated as an obstacle to interpretation—is functional. Across the first eight layers of a 12-layer GPT-2-class model, every sublayer writes into a subspace held near-orthogonal to the unembedding read-out, and rotating attention's values onto the read-out is $64$–$84\times$ more damaging than a matched random rotation, with the damage entirely in cross-token mixing. The answer is then written on-axis, late, by addition rather than by turning accumulated content. The paper further shows that the off-axis frame is a free gauge that can be prescribed: inserting one fixed rotation at the phase boundary turns a loss-only prescription that converges two seeds in eight into one that converges seven in eight (dense) or nine in nine (sparse), at baseline quality. If true, the geometry is not a curiosity; it is a mechanism that protects composition from vocabulary blur, and a control surface for training.

What carries the argument

The argument runs on four linked instruments. (1) The read-out axis: the direction(s) the tied unembedding reads, the fixed reference every angle is measured against. (2) The write-angle measurement: each sublayer's residual write is sent through the final layer-norm and tied unembedding (the logit lens) and its angle to the final logits computed; a value near $90^\circ$ means the write lives in a subspace the unembedding discards. (3) The insulation experiment: a small rotation of attention's values toward the read-out, compared against a matched random rotation and against the same rotation with cross-token averaging removed, isolating the cost of mixing on-axis values. (4) The frame device: a fixed orthogonal rotation (dense Haar quarter-turn or sparse signed permutation) inserted once at the concept/token boundary, which hands the model an off-axis frame instead of requiring it to grow one under loss pressure. Two supporting measurements carry the load: the participation ratio counts the effective dimensionality of the workspace, and the orthogonal Procrustes decomposition separates rotation from freshly written content across a span of layers.

What would settle it

Recompute the layer-by-layer write angles and the commit-span Procrustes split using per-layer learned translators (the tuned lens) instead of the frozen final decoder on layers 1–7; if early writes then point substantially toward the answer (angles far below $75$ degrees) or the commit $R^2$ rises well above $0.38$, the off-axis insulation and additive-commit claims would be artifacts of the read-out rather than properties of the model.

Watch

Extended reading notes

Core claim

On a 12-layer GPT-2-class transformer trained on OpenWebText, the paper claims, computation splits into two phases with a functional geometry. Through the concept phase (roughly layers 0–8), every sublayer writes into a subspace held near-orthogonal to the unembedding read-out: attention's writes sit $75$–$96^\circ$ off it at every depth, and feed-forward writes about $84^\circ$ until the last four layers. The off-axis position is not incidental: rotating attention's values onto the read-out is $64$–$84\times$ more damaging to next-token loss than a matched random rotation, and a matched control that removes cross-token averaging collapses the asymmetry to about $1\times$. The paper reads this as insulation: attention can average values across token positions without blurring vocabulary predictions, because the averaging happens in a subspace the unembedding discards. The answer is assembled additively and late—over the commit span a best-fit rotation explains only $R^2 = 0.38$ of the change, $64\%$ of the committed state is newly written content, the residual norm grows by $1.74\times$, and the final prediction first becomes the top decoded token at layer $9.2$ on average. The paper further claims the frame is a free gauge: $25$ runs across $13$ distinct $90^\circ$ rotations reach the same quality, two baselines from different seeds hold concepts in frames about $90^\circ$ apart while agreeing on their read-outs, and a basis drawn at random and prescribed before training is adopted with quality unchanged. Supplying a fixed rotation at the phase boundary—dense Haar or sparse signed permutation—makes the prescribed geometry reliably trainable: convergence under the loss-only step schedule rises from $2/8$ seeds to $7/8$ (dense) or $9/9$ (sparse) at baseline quality, while forcing every layer on-axis, as early-exit training does, matches perplexity, LAMBADA, and BLiMP but collapses the concept-phase workspace from about $25$ to $14$ effective dimensions.

Load-bearing premise

The angle measurements assume that reading every layer's write through the model's final output decoder tells you where that write points, even in early and middle layers where the paper itself says direct decoding shows little; if early writes are only legible through per-layer translators, the measured off-axis angles could be an artifact of the decoder's blind spot.

Editorial extensions

If this is right

  • A read-out-alignment penalty can force a transformer on-axis, but the resulting model writes the answer by near-cancellation: feed-forward contributes $+9103$ to the answer logit while attention writes $-8113$ against it, so on-axis quality is real but the model is a caution, not a solution.
  • Training for early exit (pressing every layer onto the read-out) can match perplexity, LAMBADA, and BLiMP while halving the concept-phase effective dimensionality from about $25$ to $14$; benchmarks alone would not detect the change.
  • Prescribing an off-axis/on-axis two-phase schedule through the loss alone is a lottery ($2/8$ seeds converge), while inserting one fixed rotation at the boundary makes the same schedule land reliably ($7/8$ dense, $9/9$ sparse) at baseline quality.
  • The $90^\circ$ frame is a free gauge: thirteen distinct rotations across $25$ runs yield the same model, two same-recipe baselines land in near-orthogonal concept frames, and a randomly drawn prescribed basis is adopted without quality loss.
  • If the geometry is what the paper claims, the off-axis position is a control surface: models can be trained into a chosen frame, making intermediate representations directly comparable across runs and across models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The insulation account predicts that increasing attention's mixing pressure (more heads, higher temperature, or longer contexts) should push the converged geometry closer to $90^\circ$; a seed sweep varying mixing strength would test this without new machinery.
  • The gauge-freedom result suggests a cheap comparability trick the paper does not fully exploit: training a family of models into one prescribed basis would let differences in concept geometry be read directly as differences in computation, rather than being aligned away post hoc.
  • The sparse-device convergence edge ($9/9$ vs $5/9$) points beyond interpretability: fixing an absorbable signed-permutation rotation at mid-depth may serve as a general training-stabilization intervention for transformers, testable at larger scales where from-scratch divergence rates are higher.
  • The invisible workspace collapse under early-exit pressure raises a testable concern: models that match quality on next-token benchmarks while losing half their effective dimensionality may degrade on tasks that use the removed directions, such as long-range dependency or compositional generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper trains 12-layer GPT-2-small-class models on OpenWebText and studies the geometry of the residual stream relative to the tied unembedding. It reports that attention writes 75–96 degrees off the read-out at every depth, that rotating attention values toward the read-out is 64–84 times more damaging than a matched random rotation only when cross-token mixing is active, and that the final prediction is committed late by additive writes rather than by rotation of accumulated off-axis content. It then shows that forcing every layer on-axis reaches benchmark parity while reducing the concept-phase participation ratio, that prescribing the off-axis/on-axis phase structure through a loss is a lottery (2/8 seeds), and that inserting a fixed rotation at the phase boundary makes the prescription reliable (7/8 dense, 9/9 sparse) without changing quality. Finally it presents evidence that the choice of 90-degree frame is a free gauge: different device rotations produce the same quality, two baselines' concept frames are nearly orthogonal, and a randomly prescribed basis is adopted during training.

Significance. If the central claims hold, the paper reframes off-axis intermediate representations from an obstacle into a functional insulation mechanism and demonstrates that the off-axis frame is a prescribable gauge. The experimental design is stronger than is typical: matched random-rotation controls, an inserted-rotation control that recovers R^2=1.00, foreign-basis controls, explicit convergence-rate reporting, and candid limitation statements in Section 9.4. The paper also identifies that effective-dimensionality changes are invisible to standard benchmarks, which is a useful methodological point. The main risks are instrument validity and reproducibility: the headline angle measurements use the logit lens despite the paper's own statement that it fails before the last layers, a key intervention's target direction is underspecified, and the L_frame coefficient is missing. No code or data release is mentioned.

major comments (4)
  1. [Section 2.1 and Section 10] The write-angle statistic is defined by sending each sublayer write through the final LayerNorm and tied unembedding (the logit lens), and the headline claim that attention writes 75–96 degrees off the read-out at every depth (Figure 2a) comes from this statistic. However, Section 10 states that the logit lens 'fails on early and middle layers' and names the tuned lens as 'our reading instrument,' and Table 1 lists the tuned lens as the instrument for reading the concept phase. No calibration of the two instruments is reported for the write-angle measurement. If early- and mid-depth writes live in per-layer bases that the frozen final unembedding projects into or near its null space, the reported angles could be an artifact of the decoder rather than a property of the writes. Please either justify why the logit lens is the correct instrument for the angle-to-read-out claim (as opposed to reading latent predictions), or report the same angles under the tuned lens and show they agree.
  2. [Section 3.1, Figure 3] The insulation experiment is the load-bearing evidence for the paper's central functional claim, but the target direction used to 'rotate each attention head's value vectors toward the read-out' is never specified. It is not stated whether the target is the unembedding row of the next token, the direction of the final logits at that position, a projection of the value vector through the logit lens, or a global read-out subspace. The interpretation of the 64–84 times damage ratio as showing that read-out-aligned values are dangerous only when mixed depends on this construction, and the matched random-rotation control cannot compensate for an underspecified target. Please give the exact perturbation rule, including how the angle and axis are computed at each position and head.
  3. [Section 8.3, Eq. (8)] The frame-prescription experiment does not report the coefficient with which L_frame is added to the cross-entropy objective, despite Section 2's statement that all loss terms are added with a fixed coefficient. The claim that a randomly drawn basis is adopted with quality unchanged cannot be reproduced, and its sensitivity to the coefficient cannot be assessed, without this value. Please report the coefficient and, ideally, a small sweep or at least the convergence/failure count for the chosen value.
  4. [Sections 3.2–3.3] The object whose motion is measured as the 'concept constellation' is never defined. The rigidity rotation is fitted to token residual states, but the text reports R^2=0.93 for carrying 'the whole concept constellation' from layer to layer and a 29-degree turn per layer. Without an explicit definition of the concept directions or the concept constellation, these numbers are statements about token residual clouds, and the inference that 'the frame the concept phase computes in' rotates is not supported as stated. Please define the concept directions used in the Procrustes/rigidity analysis, or rephrase the claims in terms of the token residual states that were actually measured.
minor comments (5)
  1. [Section 6.1] The sentence 'because it is orthogonal it preserves the residual norm and the participation ratio exactly' is true only at the instant of application; after training, the dense-device models show a concept-phase PR near 54 versus 24 for the baseline. Please reword to avoid implying the trained model's PR is preserved.
  2. [Table 4 and Figure 15] The table caption and figure text describe the concept-phase participation ratio as an average over layers 0–6, but Equation 4 defines the concept phase as layers 0–5 with the boundary at layer 6. Please resolve this inconsistency and state which layer set is used in the reported PR means.
  3. [Section 8.1] The statement 'Across all twenty-five, LAMBADA spans...' followed by 'Two further dense runs are set aside' is ambiguous as to whether the twenty-five includes the set-aside runs; please state the total number of runs, the number excluded, and the reason for exclusion in one place.
  4. [Section 4.3] The coefficient used for the angle constraint (Equation 3) is not stated; 'the same coefficient' refers back to a sweep reported for the distributional penalty, but the reader cannot tell which value was used for the fifteen-seed experiment. Please state it explicitly.
  5. [Reproducibility] No code or data release is mentioned; given the number of bespoke interventions (rotations, losses, schedules, screening criteria), a public release of the training and analysis code would substantially strengthen the paper.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central geometric claims are empirical, measured against matched nulls, with the functional account explicitly labeled as inference.

full rationale

The paper's derivation chain is self-contained. Write angles are defined against the fixed tied unembedding (Section 2.1), the insulation experiment compares read-out-aligned rotations against matched random rotations of the same magnitude and a no-mixing control (Section 3.1), and the additive-commit conclusion uses orthogonal Procrustes with an inserted-rotation control recovering R2_rot = 1.00 (Section 3.2). The rigidity rotation is fitted to the token stream and judged against a no-motion null (Sections 3.2-3.3). The device and frame-prescription results are explicitly training targets or fixed interventions; the non-trivial content is the empirical convergence and quality invariance, not the realization of the prescribed geometry. The only by-construction element is the manipulation check that a loss explicitly minimizing off-diagonal energy in B produces low off-diagonal energy in B; the paper's non-trivial claim there is that quality is unchanged, which is empirical. The single self-citation [35] only contextualizes benchmark levels and is not load-bearing. Section 9.4 explicitly states that the functional-insulation account is an inference from the causal measurement rather than an independent result, further reducing circularity risk. The logit-lens/tuned-lens discrepancy raised by the skeptic is a measurement-validity concern about the read-out instrument, not a case of a claimed result reducing to its input by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the target claim in a derivation sense; the fitted rotation in Section 3.2 is a measurement tool judged against a null. The hand-chosen penalty coefficients and phase boundary are training design choices, and the L_frame coefficient is unspecified. No new entities are introduced; the device is an orthogonal rotation intervention, not a new model component.

free parameters (5)
  • Read-out alignment penalty coefficient alpha for L_KL and L_cos = Swept 0.1 to 10; primary reported runs at coefficient 3
    Hand-chosen loss weight; the geometry-quality tradeoff and the cancellation result depend on it.
  • Frameless step angle schedule theta_l = 90/0 degrees = 90 degrees for layers 0-5, 0 degrees for layers 6-11
    Prescribed target schedule in Equation 4; all device and frame runs use this schedule.
  • Phase boundary layer = 6
    The concept/token split is placed at layer 6 by hand, while the baseline's own commit occurs near layer 9. The choice is not derived from the measured geometry.
  • L_frame coefficient for basis-diagonality loss = Not stated
    Equation 8 is described as added with a fixed coefficient held constant through training, but no value is reported. Exact replication is impossible without it.
  • Concept-phase layer set for participation-ratio averages = Layers 0-6
    PR means are reported over layers 0-6; the boundary at layer 6 makes this the concept phase by definition.
assumptions (4)
  • domain assumption Residual stream is additive: each block contributes a write to a shared residual, and these writes can be attributed linearly to final logits via a shared final normalization.
    Section 2.1 and Figure 6 use this view; it is established in prior work but central to all angle and attribution measurements.
  • domain assumption The final layer-norm and tied unembedding, applied directly to intermediate states, give a valid angular read-out to the vocabulary axis.
    Section 2.1 defines write angles through this logit-lens operation. The paper's own caveat that the logit lens resolves little before late layers is not resolved for these angles.
  • standard math For non-Gaussian residual states, the best linear predictor of a layer's write is a valid stand-in for the layer's linear response; Stein's lemma connects it to the mean Jacobian only under Gaussian input.
    Section 3.3 uses this distinction and cites classical results; it explains the rigid frame turn but relies on non-Gaussianity of residual states.
  • domain assumption Orthogonal Procrustes with a no-motion null is sufficient to distinguish rotational from additive change in residual states.
    Section 3.2 uses R2_rot against a no-motion baseline; an inserted-rotation control verifies the instrument detects an explicit rotation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So." pith.science (2026). https://pith.science/paper/XFJSBDYJ

@misc{pith2026260810251,
  author       = {Pith},
  title        = {Pith review of: Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFJSBDYJ}},
  note         = {Machine review of arXiv:2608.10251}
}
read the original abstract

A transformer's answer lives on one axis: the direction its unembedding reads. Its intermediate states largely do not, and that off-axis position is usually treated as an obstacle to interpretation. We show it is functional. A 12-layer model computes in two phases. Through the first, every sublayer writes into a subspace held near-orthogonal to the read-out, attention 75 to 96 degrees off it at every depth. Moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation, and the damage is entirely in cross-token mixing: the subspace insulates composition from the vocabulary. Beneath it the frame itself turns rigidly with depth. In the second phase the answer arrives on-axis, late, and by addition rather than by turning accumulated content onto the read-out. Pressing every layer onto the read-out instead, as training for early exit does, matches the baseline on perplexity, LAMBADA and BLiMP while cutting the concept-phase workspace from about twenty-five effective dimensions to fourteen, a change none of those benchmarks register. The geometry can also be imposed, though not by asking for it. Prescribing it through the loss is a lottery: six of eight seeds collapse, because a model told to null its read-out projection obeys most cheaply by discarding dimensions. Inserting one fixed rotation at the phase boundary lands it instead, at baseline quality. A sparse rotation the surrounding weights can absorb converges on all nine seeds, against five of nine for ordinary training. Which rotation is immaterial: twenty-five runs across thirteen distinct ones reach the same quality, and two baselines from different seeds hold their concepts in near-orthogonal frames while agreeing on their read-outs. That freedom is usable: a basis drawn at random and prescribed before training is adopted across the concept phase, with quality unchanged.

Figures

Figures reproduced from arXiv: 2608.10251 by the authors.

Figure 1
Figure 1. states the geometric answer in outline. Through the body of the network the com￾putation lives in a subspace held roughly orthogonal to the vocabulary read-out, attention mixes concept ingredients across token positions there, never turning them toward the output axis, and only in the last few layers does the feed-forward path write the prediction onto that axis. The answer is assembled at the read-out late; it is n… view at source ↗
Figure 2
Figure 2. Per-layer geometry of the five converged baselines (thin lines individual runs, bold their mean). (a) The angle of each sublayer’s write to the read-out: attention (blue) stays ≈83◦ off-axis at every depth, the insulator, while the feed-forward write (orange) swings toward the read-out over the last four layers, the commit. (b) The residual participation ratio: moderate through the body of the network and expanding … view at source ↗
Figure 3
Figure 3. Off-axis positioning is functional insulation for cross-token mixing (five converged baselines, each against four draws of the control rotation). Rotating attention values toward the read-out and letting attention mix them is 64× more damaging to the next-token loss than a matched random rotation of the same magnitude at a 10◦ rotation, and 84× at 20◦ (blue); removing the cross-token averaging, each position keeping… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: The commit is written, not rotated (mean over the five converged runs). (a) A best-fit rotation explains only a minority of each transition (R2 rot, blue), while most of the resulting state is freshly written content (orange); an explicitly inserted rotation is recover…
Figure 5
Figure 5. Figure 5: The prediction surfaces late (mean over the five converged runs). For each token position, the first layer at which the logit-lens top token matches and holds the final prediction; the mass sits in layers 8–11, with 39% of positions resolving only at the final layer an…
Figure 6
Figure 6. Figure 6: How the answer is assembled (direct logit attribution with a shared final normalization, so the sublayer writes sum to the logits exactly; median over the five converged runs). (a) Each sublayer’s contribution to the final-answer logit: the last feed-forward layers wri…
Figure 7
Figure 7. Figure 7: The geometry of a usable model, and of the ones that fail (five converged, four unusable). (a) Residual participation ratio: the converged runs (band) hold a moderate-then-expanding profile, while the four failed runs collapse below five, inflate past one hundred thirt…
Figure 8
Figure 8. Figure 8: The pressure applied in this section. Every layer’s decode is asked to point along the final prediction: a target angle of 0 ◦ at all twelve depths, with nothing inserted into the residual stream. Each arrow is the prescribed target for that layer, measured from the re…
Figure 9
Figure 9. Figure 9: Forcing the computation on-axis with the read-out-alignment penalty, swept in coefficient. (a) The mid-stack residual angle to the read-out falls with pressure and saturates toward a ≈ 13◦ floor. (b) LAMBADA accuracy falls and validation perplexity climbs across the sa…
Figure 10
Figure 10. Figure 10: How the on-axis-forced model reaches the answer, by direct logit attribution (symmetric-log scale; note the different vertical scales). This is the hard 0 ◦ angle target at coefficient 3, averaged over its six converged seeds (near-baseline quality, LAMBADA 0.258 / pe…
Figure 11
Figure 11. Figure 11: The pressure applied in this section. The same angle penalty as Section 4, but with a per-layer target that encodes two phases rather than one on-axis goal: 90◦ through L0–L5 and 0 ◦ through L6–L11 (Equation 4). Nothing is inserted into the residual stream, so the bou…
Figure 12
Figure 12. Figure 12: The frameless step realizes the prescribed geometry but strains before the boundary. (a) Residual angle to the read-out across depth: the baseline eases onto the axis over its final layers, while both completed step seeds hold ≈90◦ through the concept phase and turn ≈…
Figure 13
Figure 13. Figure 13: Among the seeds that converge, the concept-phase workspace either holds or collapses. Residual participation ratio across depth (log scale) for the baseline and the two completed step seeds. The survivor (orange) expands its workspace through the concept phase as the …
Figure 14
Figure 14. Figure 14: The pressure applied in this section, and the device. The schedule is unchanged from [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: The device lands the geometry and keeps the room (converged dense-device flock; band spans the flock, bold the mean). (a) Residual angle to the read-out across depth: the concept phase holds ≈ 90◦ and turns onto the read-out at the boundary, over the whole converged s…
Figure 16
Figure 16. Figure 16: Two motions, told apart, and both survive the device. (a) In the baseline every layer lands on a frame of its own: one rigid rotation carries the whole constellation across each transition (R2 = 0.93 for a rotation fitted only to the token stream, against 0.76 for no …
Figure 17
Figure 17. Figure 17: The form of the frame changes the geometry, not the value. Concept-phase participation ratio across depth: the sparse device (orange) holds the workspace at about half the dimensionality of the dense device (blue), near the baseline (grey), while both realize the same…
Figure 18
Figure 18. Figure 18: The 90◦ frame is free, and it can be prescribed exactly. (a) Twenty-five device runs, spanning thirteen distinct devices with the remainder training-seed replicates of one of them, set against the nine baseline seeds themselves. Every device run lands in the converged…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 26 canonical work pages

  1. [1]

    Three-Phase Transformer

    Mohammad R. Abu Ayyash. Three-phase transformer.arXiv preprint arXiv:2604.14430, 2026. 33

    work page Pith review arXiv 2026

  2. [2]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2401.15024

    arXiv 2024

  3. [3]

    Converting mlps into polynomials in closed form.arXiv preprint arXiv:2502.01032, 2025

    Nora Belrose and Alice Rigg. Converting mlps into polynomials in closed form.arXiv preprint arXiv:2502.01032, 2025

    arXiv 2025

  4. [4]

    Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112, 2023

    arXiv 2023

  5. [5]

    An Analysis of Residual-Stream Geometry Across Transformer Depth

    Sunit Bhattacharya and Ravi Shankar Kolli. An analysis of residual-stream geometry across transformer depth.arXiv preprint arXiv:2607.18348, 2026

    work page Pith review arXiv 2026

  6. [6]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, et al. Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Cir- cuits Thread, 2023.https://transformer-circuits.pub/2023/monosemantic-features/ index.html

    work page 2023

  7. [7]

    Adam optimizer causes privileged basis in transformer lan- guage model residual stream

    Diego Caples and rrenaud. Adam optimizer causes privileged basis in transformer lan- guage model residual stream. LessWrong, 2024.https://www.lesswrong.com/posts/ yrhu6MeFddnGRSLtQ/adam-optimizer-causes-privileged-basis-in-transformer-lm

    work page 2024

  8. [8]

    PaLM: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240): 1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, PaulBarham, HyungWonChung, CharlesSutton, SebastianGehrmann, etal. PaLM: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240): 1–113, 2023. arXiv:2204.02311

    arXiv 2023

Show all 57 references
  1. [9]

    A bias bound for least squares linear regression.Statistica Sinica, 1(1):127–136, 1991

    Naihua Duan and Ker-Chau Li. A bias bound for least squares linear regression.Statistica Sinica, 1(1):127–136, 1991

    1991

  2. [10]

    A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021.https://transformer-circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021.https://transformer-circuits. pub/2021/framework/...

    2021

  3. [11]

    Toy models of superposition.Transformer Circuits Thread, 2022.https:// transformer-circuits.pub/2022/toy_model/index.html

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, et al. Toy models of superposition.Transformer Circuits Thread, 2022.https:// transformer-circuits.pub/2022/toy_model/index.html

    2022

  4. [12]

    Privileged bases in the transformer residual stream

    Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023.https://transformer-circuits.pub/ 2023/privileged-basis/index.html

    2023

  5. [13]

    Layerskip: Enabling early exit inference and self-speculative decoding

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decoding. InProceedi...

    2024 arXiv

  6. [14]

    Dynamics of the transformer residual stream: Coupling spectral geometry to network topology.arXiv preprint arXiv:2605.14258, 2026

    Jesseba Fernando and Grigori Guitchounts. Dynamics of the transformer residual stream: Coupling spectral geometry to network topology.arXiv preprint arXiv:2605.14258, 2026. 34

    2026 arXiv

  7. [15]

    A theory of multineuronal dimensionality, dynamics and measurement

    Peiran Gao, Eric Trautmann, Byron M Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. bioRxiv preprint 214262, 2017

    2017

  8. [16]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. arXiv:2012.14913

    2021 arXiv

  9. [17]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

    Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. arXiv:2203.14680

    2022 arXiv

  10. [18]

    Openwebtext corpus.http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.http://Skylion007.github.io/ OpenWebTextCorpus, 2019

    2019

  11. [19]

    Verbalizable representations form a global workspace in language models.Transformer Circuits Thread, 2026.https://transformer-circuits.pub/2026/workspace/, ICLR 2026

    Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, et al. Verbalizable representations form a global workspace in language models.Transformer Circuits Thread, 2026.https://transformer-circuits.pub/2026/workspace/, ICLR 2026

    2026

  12. [20]

    The concept allocation zone: Tracking how concepts form across transformer depth.arXiv preprint arXiv:2605.24856, 2026

    James Henry. The concept allocation zone: Tracking how concepts form across transformer depth.arXiv preprint arXiv:2605.24856, 2026

    2026 arXiv

  13. [21]

    Geometric evolution maps: Extracting stable concept probes from transformer residual streams.arXiv preprint arXiv:2605.25848, 2026

    James Henry. Geometric evolution maps: Extracting stable concept probes from transformer residual streams.arXiv preprint arXiv:2605.25848, 2026

    2026 arXiv

  14. [22]

    Decomposing representation space into interpretable sub- spaces with unsupervised learning

    Xinting Huang and Michael Hahn. Decomposing representation space into interpretable sub- spaces with unsupervised learning. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2508.01916; ICLR 2026; preprint arXiv:2508.01916

    2026 arXiv

  15. [23]

    Residual connections encourage iterative inference

    Stanisław Jastrzębski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. InInternational Conference on Learning Representations (ICLR), 2018

    2018

  16. [24]

    Unraveling token prediction refinement and identifying essential layers in language models.arXiv preprint arXiv:2501.15054, 2025

    Jaturong Kongmanee. Unraveling token prediction refinement and identifying essential layers in language models.arXiv preprint arXiv:2501.15054, 2025

    2025 arXiv

  17. [25]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97 ofPMLR, pages 3519–3529, 2019. arXiv:1905.00414

    2019 arXiv

  18. [26]

    The remarkable robustness of LLMs: Stages of inference? InarXiv preprint arXiv:2406.19384, 2024

    Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of LLMs: Stages of inference? InarXiv preprint arXiv:2406.19384, 2024

    2024 arXiv

  19. [27]

    To FP8 and back again: Quantifying reduced precision effects on LLM training stability.arXiv preprint arXiv:2405.18710, 2024

    Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. To FP8 and back again: Quantifying reduced precision effects on LLM training stability.arXiv preprint arXiv:2405.18710, 2024

    2024 arXiv

  20. [28]

    Regression analysis under link violation.The Annals of Statistics, 17(3):1009–1052, 1989

    Ker-Chau Li and Naihua Duan. Regression analysis under link violation.The Annals of Statistics, 17(3):1009–1052, 1989

    1989

  21. [29]

    Optimal degrees of synaptic connectivity.Neuron, 93(5):1153–1164, 2017

    Ashok Litwin-Kumar, Kameron Decker Harris, Richard Axel, Haim Sompolinsky, and L F Abbott. Optimal degrees of synaptic connectivity.Neuron, 93(5):1153–1164, 2017. 35

    2017

  22. [30]

    A geometric perspective on next-token prediction in large language models: Three emerging phases.arXiv preprint arXiv:2605.09011, 2026

    Gianfranco Lombardo, Giuseppe Trimigno, and Stefano Cagnoni. A geometric perspective on next-token prediction in large language models: Three emerging phases.arXiv preprint arXiv:2605.09011, 2026

    2026 arXiv

  23. [31]

    Residual connections and the causal shift: Uncovering a structural misalignment in transformers.arXiv preprint arXiv:2602.14760, 2026

    Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Axel Marmoret, Lukas Mauch, Fabien Cardinaux, and Ghouthi Boukli Hacene. Residual connections and the causal shift: Uncovering a structural misalignment in transformers.arXiv preprint arXiv:2602.14760, 2026

    2026

  24. [32]

    Morcos, Maithra Raghu, and Samy Bengio

    Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. InAdvances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  25. [33]

    Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth

    Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations (ICLR), 2021. arXiv:2010.15327

    2021 arXiv

  26. [34]

    Interpreting GPT: The logit lens

    nostalgebraist. Interpreting GPT: The logit lens. LessWrong, 2020.https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  27. [35]

    Legible-by-construction: Attention and end-to-end transformers.arXiv preprint arXiv:2607.04319, 2026

    Mark Oskin. Legible-by-construction: Attention and end-to-end transformers.arXiv preprint arXiv:2607.04319, 2026

    2026 arXiv

  28. [36]

    EE-Tuning: An eco- nomical yet scalable solution for tuning early-exit large language models.arXiv preprint arXiv:2402.00518, 2024

    Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, and Jingren Zhou. EE-Tuning: An eco- nomical yet scalable solution for tuning early-exit large language models.arXiv preprint arXiv:2402.00518, 2024

    2024 arXiv

  29. [37]

    The LAM- BADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAM- BADA dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th Annual Meeting of ...

    2016

  30. [38]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. InProceedings of the 41st International Conference on Machine Learning (ICML), volume 235 ofPMLR, pages 39643–39666, 2024. arXiv:2311.03658

    2024 arXiv

  31. [39]

    Language models are unsupervised multitask learners.OpenAI Technical Report, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI Technical Report, 2019

    2019

  32. [40]

    SVCCA: Singu- lar vector canonical correlation analysis for deep learning dynamics and interpretability

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singu- lar vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  33. [41]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. InAnnual Meeting of the Asso- ciation for Computational Linguistics (ACL), 2024. arXiv:2312.06681

    2024 arXiv

  34. [42]

    The effective rank: A measure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference (EUSIPCO), pages 606–610. IEEE, 2007

    2007

  35. [43]

    A generalized solution of the orthogonal Procrustes problem.Psychome- trika, 31(1):1–10, 1966

    Peter H Schönemann. A generalized solution of the orthogonal Procrustes problem.Psychome- trika, 31(1):1–10, 1966. 36

    1966

  36. [44]

    Tran, Yi Tay, and Donald Metzler

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. InAdvances in Neural Informa- tion Processing Systems 35 (NeurIPS), pages 17456–17472, 2022. arXiv:2207.07061

    2022 arXiv

  37. [45]

    Confidence regulation neurons in language models

    Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.16254

    2024 arXiv

  38. [46]

    RoFormer: Enhancedtransformerwithrotarypositionembedding.arXiv preprint arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhancedtransformerwithrotarypositionembedding.arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  39. [47]

    Signed-permutation coordinate transport for RMSNorm transformers.arXiv preprint arXiv:2606.31963, 2026

    John Sweeney. Signed-permutation coordinate transport for RMSNorm transformers.arXiv preprint arXiv:2606.31963, 2026

    2026 arXiv

  40. [48]

    Alex Tamkin, Mohammad Taufeeque, and Noah D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks. InProceedings of the 41st International Conference on Machine Learning (ICML), volume 235 ofPMLR, pages 47535–47563, 2024. arXiv:2310.17230

    2024 arXiv

  41. [49]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without opti- mization.arXiv preprint arXiv:2308.10248, 2023

    2023 arXiv

  42. [50]

    PolyPythias: Stabilityandoutliersacrossfiftylanguage model pre-training runs

    Oskar van der Wal, Pietro Lesci, Max Müller-Eberstein, Naomi Saphra, Hailey Schoelkopf, WillemZuidema, andStellaBiderman. PolyPythias: Stabilityandoutliersacrossfiftylanguage model pre-training runs. InInternational Conference on Learning Representations (ICLR),

  43. [51]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2017

    2017

  44. [52]

    BLiMP: The benchmark of linguistic minimal pairs for English

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics (TACL), 8:377–392, 2020

    2020

  45. [53]

    Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D

    Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl- Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for larg...

    2024 arXiv

  46. [54]

    Jump to conclusions: Short-cutting transformers with linear transformations

    Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. InProceedings of the 2024 Joint In- ternational Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)...

    2024 arXiv

  47. [55]

    Haeffele, and Yi Ma

    Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Ben- jamin D. Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.01129. 37

    2023 arXiv

  48. [56]

    Haeffele, and Yi Ma

    Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Druv Pai, Benjamin D. Haeffele, and Yi Ma. White-box transformers via sparse rate reduction: Compres- sion is all there is?Journal of Machine Learning Research (JMLR), 2024. arXiv:2311.13110

    2024 arXiv

  49. [57]

    OPT: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022. 38

    2022 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.