Pith. sign in

REVIEW 4 major objections 4 minor 4 references

In decoder-only LLMs, a sharp transition at two-thirds depth separates context processing from prediction formation, and late-layer angular geometry—not vector norm—causally governs next-token choice.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:35 UTC pith:IJB767ZZ

load-bearing objection New late-layer angular-code observation is worth following, but the phase-transition claim is confounded by readout distance. the 4 major comments →

arxiv 2602.04931 v2 pith:IJB767ZZ submitted 2026-02-04 cs.LG cs.AI

Emergent Causal-Geometric Dynamics Across Depth in Large Language Models

Shahar Haim , Daniel C McNamee This is my paper
classification cs.LG cs.AI MSC 68T0768T50
keywords depth-wise dynamicsrepresentation geometrycausal interventionphase transitionangular codingprediction-formingcontext-processingdecoder-only LLMs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that decoder-only LLMs organize their computation into two depth-wise phases: an early context-processing phase and a late prediction-forming phase, with a sharp transition at roughly two-thirds of the layers. Within the late phase, the angular (directional) geometry of token representations parameterizes similarity between next-token output distributions, whereas representation norms encode context-specific information that does not determine which token is predicted. The authors support this by unifying geometric analyses with causal interventions: perturbing the direction of the final token's representation in late layers selectively changes predictions, while norm-based perturbations do not, and input-token perturbations matter only in early layers. If correct, this reconciles geometric and intervention-based accounts of LLM internals and implies that reliable control of LLM outputs should target late-layer angular degrees of freedom.

Core claim

The central discovery is a two-component geometric code localized to the prediction-forming phase of decoder-only LLMs. Across three 7-8B models (Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B), the paper shows that the similarity between two tokens' output distributions is parameterized by the angular distance between their late-layer representations, while Euclidean distance—dominated by norm fluctuations—is orthogonal to predictive similarity. Purely angular steering of the final token remains potent in the last third of layers; purely norm-based steering fails everywhere. The mechanism is traced to the softmax readout, where a representation's direction sets the rank order of logits while its nor

What carries the argument

Two instruments carry the argument. First, a pair of layer-wise causal interventions: an input-centric steering vector, computed as a centroid difference between input-token representations, is added to an input token; a prediction-centric steering vector, computed from final-token representations grouped by model prediction, is added to the final token. The crossover of their effect profiles across layers locates the phase transition. Second, the readout decomposition p = Softmax(W_out ||h||_2 ĥ) for a final-token representation h separates the direction ĥ, which determines relative logits, from the norm ||h||_2, which acts as an inverse-temperature-like global scale. These are combined wit

Load-bearing premise

The load-bearing premise is that the depth at which prediction-centric interventions on the final token become effective marks a genuine computational phase rather than an artifact of the final token's representation being the direct input to the unembedding head, which makes late-layer directional changes trivially more effective than input-token perturbations that must propagate through remaining layers.

What would settle it

Take LLM families with very different depths (e.g., 12, 32, 70 layers) and locate the intervention-crossover boundary on a fixed task. If the boundary occurs at a constant number of layers above the output head, or if reading out from an earlier layer with a logit lens moves the boundary correspondingly, the two-phase interpretation is an artifact of readout proximity; if the boundary stays at a constant fraction of depth, the phase interpretation survives. A complementary check: attach a temporary linear readout to an early-layer final-token representation—if output-centric angular steering b

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Late-layer angular geometry is a causally privileged control variable: rotating the final token representation in the last third of layers reliably changes the predicted token, while norm changes do not.
  • Probe-decodable structure is not sufficient for control: norms and early-layer angles may encode information that is causally inert, so interpretability claims need causal validation.
  • Layer-wise analysis of LLMs must be embedded in global depth-wise dynamics; the two-phase structure provides a principled layer range for model editing and steering.
  • Angular similarity as a semantic-similarity measure is mechanistically grounded specifically in prediction-forming layers, not uniformly across depth.
  • The sharp phase boundary offers a target for evaluation: intervention strategies should be assessed on causally operative geometry, not just linear separability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The readout-decomposition argument suggests a testable architectural prediction: models with stronger normalization (e.g., RMSNorm variants) should exhibit even more pronounced angular dominance because norms are compressed; ablating normalization layers should weaken the dissociation between angle and norm.
  • If the two-phase structure is generic, then early-exit or logit-lens methods should find that representations become 'predictive' in angular terms only after the transition; this could yield a principled rule for choosing which layer to read out for downstream tasks.
  • The framework may extend beyond single-token prediction to multi-step reasoning: intermediate state evolution in algorithmic tasks might introduce additional phase boundaries or shift the transition depth, which could be tested by repeating the intervention analysis on tasks with multi-token outputs.
  • The norm's role as a context-carrying but prediction-decoupled signal hints that representation norm could serve as a diagnostic of contextual difficulty or distributional uncertainty without altering token identity; measuring norm against prediction entropy across tasks would test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a depth-wise organization of computation in decoder-only LLMs: an early 'context-processing' phase and a late 'prediction-forming' phase, separated by a sharp transition at roughly two-thirds depth. Using intervention experiments on the Months task (input-centric vs. prediction-centric steering) and geometric analyses on wikitext-103 (participation ratio, angular/Euclidean distances vs. prediction KL divergence), it argues that late-layer angular structure parameterizes prediction-distribution similarity and enables selective causal control, while representation norms are largely decoupled from prediction. The central falsifiable prediction is that purely angular interventions are potent only in the late phase, whereas norm interventions are ineffective.

Significance. If the central claims hold, the paper would provide a valuable synthesis of geometric and causal perspectives on LLM interpretability, with practical implications for layer-specific steering and control. The use of three open-weight model families (Llama-8B, Mistral-7B, Qwen-7B) and the construction of a genuinely separate angular-intervention test are strengths. The paper also offers a clean mechanistic rationale for angular vs. norm effects via the readout decomposition (Softmax(W_out ||h_L|| \hat{h}_L)). However, the empirical support is weakened by the absence of any error bars or significance testing and by a readout-proximity confound that affects the central phase-transition claim. The angular-intervention prediction is a good start, but the current analyses do not yet establish the sharp transition or the two-phase organization with the strength claimed in the abstract.

major comments (4)
  1. [§3, Fig. 1 (and Fig. 4)] The load-bearing claim of a sharp context-to-prediction transition is not supported by a depth-matched control. The prediction-centric intervention is applied to the final token, which is the direct input to the unembedding (p=Softmax(W_out h_L)), while the input-centric intervention is applied to earlier tokens that must propagate through all remaining layers. The observed crossover is exactly what this readout-distance asymmetry would produce, independent of any representational phase. Please provide controls: e.g., apply prediction-derived vectors to the final token at early layers vs. late layers; apply the same vector type to tokens at different distances from the output; and include a random-direction baseline at each layer. Without such controls, the 'phase-change point' cannot be attributed to a computational phase.
  2. [§3, 'Angular interventions', Fig. 4] The angular-intervention result is presented as confirmation of the phase transition, but it is largely a direct consequence of the readout structure: the decomposition p=Softmax(W_out ||h_L|| \hat{h}_L) implies that changing the direction of the final-token representation will alter logits, while changing only its norm will not. The fact that output-based angular interventions are effective in late layers may therefore reflect the final token's proximity to the unembedding, not a genuinely new geometric code. To make the claim convincing, compare angular interventions on the final token at early vs. late layers (depth-matched, since the token position is identical) and include a control where angular intervention is applied to an intermediate token whose representation is not directly read out. Also report whether the effect magnitude scales with layer depth as a continuous function, or
  3. [§3, Figs. 1–4 and Spearman correlations] No error bars, confidence intervals, or significance tests are reported for the intervention curves, participation ratio profiles, or the Spearman correlations between distance and prediction divergence. The 'sharp transition' is a qualitative crossover, and the vertical phase-change line in Figs. 2–4 is taken from Fig. 1 without any independent validation on the wikitext data. Please define how the phase-change point was determined (e.g., crossing point, change-point test), report bootstrap or multi-seed variability, and provide a statistical test that the late-layer angular–prediction correlation is nonzero and significantly greater than the early-layer correlation. Without this, the central narrative of a sharp, robust transition is overstated.
  4. [Abstract and §3 (Fig. 1)] The paper consistently uses 'sharp transition' and 'two phases' to describe the intervention results, but the plotted curves in Fig. 1 appear to be smooth crossovers with no visible discontinuity. This is not merely a wording issue: the entire geometric interpretation (vertical lines in Figs. 2–4) depends on treating the crossover as a discrete boundary. If the crossover is gradual, then the two-phase language and the transfer of a single boundary layer to other datasets are not justified. At minimum, a quantitative comparison (e.g., a fitted mixture model vs. a smooth function) is needed to support the 'sharp' descriptor.
minor comments (4)
  1. [§3, first paragraph of 'Context- and token-dependent dimensionality'] The paragraph beginning 'To isolate the contribution of contextual variability...' appears twice verbatim. Please remove the duplicate.
  2. [§2.2 and Fig. 2] The term 'pilot-token representations' is used without definition. Clarify that these are the last-token (identical) and fourth-from-last-token (non-identical) sets described in the data section. Also, in Fig. 2 the 'baselining' of ordered PR by shuffled PR should be explicitly defined (signed difference? ratio?) both in the text and in the figure legend.
  3. [§3, Fig. 4 caption and text] The blue vertical line in Fig. 4 is said to mark the 'original perturbation-based phase-change point.' Since the angular intervention is a separate experiment, it would be informative to report the phase-change point estimated from the angular curves alone, and any discrepancy.
  4. [§2.1 and Fig. S1] The text refers to 'Methodology, Interventions' but the actual subsection is 'Intervention setup.' Correct the cross-reference.

Circularity Check

0 steps flagged

No significant circularity: the intervention crossover and angular-code claims are empirical findings with independent content; the readout-proximity confound is a validity concern, not a circular reduction.

full rationale

I walked the derivation chain from the intervention experiments (Fig. 1) to the phase-transition claim, then to the geometric analyses (Figs. 2-3), and finally to the angular-intervention test (Fig. 4). At no point does an equation reduce a central claim to a fitted parameter or to a self-citation. The phase-change point is estimated from intervention effect curves and then reused as a visual reference in later figures, but that is a display choice, not a definitional equivalence. The angular-intervention prediction is a separate manipulation (normalizing vectors and replacing directions) rather than a re-fit of the same data that defined the phase boundary; the paper even introduces the readout decomposition p(output)=Softmax(W_out ||h_L|| h_hat_L) after the angular results as a 'mechanistic rationale,' not as the source of the prediction. The observed correlations between angular distance and prediction KL divergence are empirical, as are the norm-intervention failures. There are no load-bearing self-citations: the cited prior work (e.g., Engels et al., Lad et al.) is external and does not supply an unverified uniqueness theorem or ansatz. The main weakness—that prediction-centric interventions operate on the final token, which is closer to the unembedding, while input-centric interventions must propagate through remaining layers—is a potential confound and a correctness/internal-validity risk, but it is not circularity under the required standard. The paper itself flags the early-to-late mechanistic link as speculative in its Limitations section, further supporting that these are open empirical questions rather than closed definitional loops.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claim rests on a small number of assumptions: a linear readout decomposition, the transfer of a task-specific phase boundary to natural-text geometry, and the validity of distance–KL correlations as a measure of predictive geometry. No new entities are introduced; one data-derived cutoff (phase-change layer) is reused across analyses.

free parameters (2)
  • Phase-change layer per model = ≈ 2/3 of layers (from Fig. 1, Months task)
    Obtained from the crossover of input- vs prediction-centric intervention curves; used as the fixed vertical cutoff in all later geometric analyses (Figs. 2–4) without error bars or a transfer check.
  • Steering vector addition scale = 1.0 (implicit)
    Interventions are applied as un-scaled vector addition ('simple vector addition', §2.1); the magnitude of the added vector strongly affects effect sizes and is not justified or swept.
axioms (4)
  • domain assumption Output distribution is Softmax(W_out h_L) with bias ignored, so direction sets relative logits and norm is an inverse temperature.
    Invoked in §3 ('Two-component geometric coding') to explain the angular/norm dissociation; exact for a linear unembedding head without bias, which holds for these models but is presented without caveat.
  • ad hoc to paper The Months-task phase boundary transfers to wikitext-103 representation analyses.
    The vertical phase-change line estimated from Fig. 1 is overlaid on PR and correlation curves computed on wikitext data (Figs. 2–4), with no test that the boundary is task-independent.
  • domain assumption Spearman correlation of pairwise angular/Euclidean distances with pairwise symmetric KL divergence of next-token distributions measures prediction-aligned geometry.
    Used in §3 to infer the two-component code; this aggregation ignores token frequency, position, and other confounds, and significance is not reported.
  • standard math Participation ratio on L2-normalized token vectors estimates effective dimensionality.
    Standard variance-decomposition measure; normalization is a modeling choice stated in §2.2, with unnormalized results deferred to the supplementary.

pith-pipeline@v1.3.0-alltime-deepseek · 11832 in / 12062 out tokens · 116015 ms · 2026-08-03T04:35:52.936316+00:00 · methodology

0 comments
read the original abstract

Geometric analyses of large language model (LLM) representations reveal structured variation across depth but remain fundamentally correlational with respect to token prediction formation. Meanwhile, causal interventions expose depth-dependent efficacy profiles without a unifying account of their representational dynamics. A complete account of LLM function requires explaining how representational structure evolves across depth to causally produce predictions. We synthesize these perspectives by combining geometric analysis with mechanistic interventions, explicitly centralizing depth-wise dynamics as the organizing axis for interpreting LLM function. In decoder-only LLMs, we identify a sharp transition from context-processing to prediction-forming computation, accompanied by a more gradual reorganization of representational geometry across layers. This synthesis reveals a late-layer geometric code in which angular structure parameterizes next-token distributional similarity and enables selective causal control over predictions, while representation norms encode information largely decoupled from prediction. Together, our results provide a synthesis of causal and geometric perspectives, yielding a mechanistic account of how control-relevant geometric dynamics across depth transform context into prediction in language models. This perspective reconciles previously puzzling findings and implies that layer-wise function cannot be understood or effectively intervened upon in isolation, but only within the emergent global dynamical structure of the network.

Figures

Figures reproduced from arXiv: 2602.04931 by Daniel C McNamee, Shahar Haim.

Figure 1
Figure 1. Figure 1: Per-layer intervention experiment results. Average logit preference difference: blue curves for input based interventions (interval and month), Red curve for output based intervention. The blue vertical line marks the phase-change point. different PR profiles for both identical and non-identical tokens, indicating that the observed patterns are learned properties of trained models rather than architectural… view at source ↗
Figure 2
Figure 2. Figure 2: Per-layer participation ratio (PR) of normalized pilot-token representations for ordered long and short sequences, with shuffled counterparts shown as baselines. Top row: identical tokens. Bottom row: non-identical tokens. Columns correspond to models (Llama, Mistral, Qwen, left to right). The blue vertical line marks the perturbation-based phase-change point. directly track predictive similarity. This lea… view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer correlation between non-identical tokens’ pairwise distances and pairwise prediction-distribution symmetric KL divergence, shown across layers. Top row: angular distances. Bottom row: Euclidean distances. Rows correspond to distance metrics, and columns correspond to models. The blue vertical line marks the perturbation-based phase-change point. zhigaev et al., 2024) and intervention-based, outpu… view at source ↗
Figure 4
Figure 4. Figure 4: Per-layer intervention experiment results. Top row: pure angular interventions. Bottom row: pure norm interventions. Average logit preference difference is shown: blue curves correspond to input-based interventions (interval and month), red curves to output-based interventions. The blue vertical line marks the original perturbation-based phase-change point. limitation of this assumption: decodable represen… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith

  1. [238]

    why should i trust you?

    URL https://aclanthology.org/2025. findings-emnlp.238/. Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models.arXiv preprint, arXiv:2311.03658, 2023. doi: 10.48550/arXiv.2311.03658. URL https://arxiv. org/abs/2311.03658. Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.03658 2025

  2. [2013]

    URL https://aclanthology.org/N13-1090/

    Association for Computational Linguistics. URL https://aclanthology.org/N13-1090/. Pan, B., Li, Y ., Zhang, W., Lu, W., Xu, M., Zhou, S., Zhu, Y ., Zhong, M., and Qian, T. A survey on training- free alignment of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2025, pp. 4445–4461. Association for Computational Lin-...

    doi:10.18653/v1/2025.findings-emnlp 2025

  3. [2018]

    cc/paper_files/paper/2018/hash/ 294a8ed24b1ad22ec2e7efea049b8737-Abstract

    URL https://proceedings.neurips. cc/paper_files/paper/2018/hash/ 294a8ed24b1ad22ec2e7efea049b8737-Abstract. html. Alain, G. and Bengio, Y . Understanding intermediate 8 Prediction-Centric Geometry in LLMs layers using linear classifier probes.arXiv preprint arXiv:1610.01644, 2016. Barbero, F., Banino, A., Kapturowski, S., Kumaran, D., Ara´ujo, J. G. M., V...

    Pith/arXiv arXiv 2018

  4. [2019]

    URL https://arxiv.org/abs/1909. 00512. Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., and Kagal, L. Explaining explanations: An overview of interpretability of machine learning.arXiv preprint, arXiv:1806.00069, 2018. URL https://arxiv. org/abs/1806.00069. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. I...

    Pith/arXiv arXiv 1909