Pith. sign in

REVIEW 4 major objections 5 minor 38 references

This paper proposes that LLM activations decompose into sparse, uniquely recoverable 'atoms' under a non-Euclidean inner product, reporting 99.9% reconstruction fidelity and 99.8% stability across three models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 15:13 UTC pith:ESDVNHHS

load-bearing objection The atom concept is circular: the metric S is built from the dictionary D it certifies, so the low coherence is generic after whitening; the empirical SAE reconstruction is plausible but doesn't bear the theoretical weight. the 4 major comments →

arxiv 2509.20784 v3 pith:ESDVNHHS submitted 2025-09-25 cs.CL cs.AI

Towards Atoms of Large Language Models

Chenhui Hu , Pengfei Cao , Yubo Chen , Kang Liu , Jun Zhao This is my paper
classification cs.CL cs.AI
keywords atoms theoryatomic inner productsparse autoencodersrepresentation shiftLLM interpretabilitysparse recoveryuniquenessthreshold activation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Atoms Theory, which defines the fundamental representational units of large language models as 'atoms': sparse directions in activation space that are approximately orthogonal under a specially chosen non-Euclidean inner product, the atomic inner product (AIP). The authors argue that the usual Euclidean geometry of activations is distorted by the softmax operation, producing a pervasive 'representation shift,' and that AIP corrects this shift. They prove that if atoms satisfy a coherence bound, they enjoy uniqueness and exact ℓ1-recoverability guarantees, and that sparse autoencoders with threshold activation can identify them. Empirically, on Gemma2-2B, Gemma2-9B, and Llama3.1-8B, the paper reports near-perfect sparse reconstruction (R²≈99.9%) and that over 99.8% of learned atoms satisfy the uniqueness condition, compared with 0.5% for neurons and 68.2% for standard SAE features. If correct, this offers a principled unit for mechanistic interpretability and a theoretical foundation for understanding how LLMs store knowledge.

Core claim

Atoms Theory asserts that each LLM representation is a sparse, nonnegative combination of a shared set of atoms, and that this decomposition is identifiable and stable. The central discovery is that there exists a non-Euclidean inner product — the atomic inner product, ⟨x,y⟩_S = x^T S y with S = (DD^T)^{-1} — under which these atoms are approximately orthogonal, making them mutually distinguishable. The paper proves that a dictionary of atoms satisfying a coherence condition µ < 1/(2K−1) guarantees unique sparse coefficients and exact ℓ1 recovery, and that a single-layer sparse autoencoder with threshold (JumpReLU) activation provably recovers the atom set. Empirical results on three open mo

What carries the argument

The atomic inner product (AIP): a symmetric positive-definite bilinear form ⟨x,y⟩_S = x^T S y with S = (DD^T)^{-1}, defined from the atom dictionary D, which the paper shows corrects the representation shift caused by softmax and makes distinct atoms approximately orthogonal. The companion mechanism is the threshold-activated sparse autoencoder (TSAE), a single-layer linear–nonlinear encoder whose JumpReLU activation with a support-separating threshold provably identifies the atom set; the proof leverages the coherence upper bound of the Restricted Isometry Property, µ ≤ ε, and the uniqueness condition µ < 1/(2K−1).

Load-bearing premise

The load-bearing premise is that the atomic inner product S=(DD^T)^{-1}, computed from the dictionary D that the paper labels as atoms, is the true geometry of LLM representations; the near-orthogonality that defines atoms is measured under a metric derived from those same atoms, so the definition is self-consistent rather than independently validated.

What would settle it

Compute the atomic inner product matrix S from a dictionary learned on one half of the activations, then test the near-orthogonality of atoms on the other half; if the NAIP distribution broadens substantially, the reported atomicity depends on fitting S to the same data and is not a property of the representation geometry.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Atoms give the first formally defined fundamental unit for LLM representations, with uniqueness and recoverability guarantees that neurons and features lack.
  • Threshold-activated SAEs (not ReLU or TopK) are the appropriate architecture for atom identification, providing a practical design rule for interpretability tools.
  • Representation shift correction via AIP should be applied before any geometric analysis of activations, since Euclidean inner products distort the underlying geometry.
  • Reliable atom recovery requires SAE capacity to match the scale of the data; beyond a critical capacity threshold, reconstruction stabilizes, suggesting a scaling law for interpretability.
  • If atoms are universal across layers and models, they could serve as a common vocabulary for cross-model and cross-layer comparisons of knowledge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the same atom set should be recoverable from held-out activations and from different random seeds; if it is not, atomicity may be an artifact of fitting S to D.
  • The AIP metric could be applied to knowledge editing and circuit analysis, where Euclidean cosine similarity is currently standard; the corrected geometry may change localisation results.
  • The comparison against features uses open-source dictionaries trained on continuous-text MLP activations, whereas atoms are trained on entity-knowledge activations; a matched training setup would be needed to attribute the stability gap to the atoms themselves.
  • If atom stability holds across domains, atoms could enable interpretability of factual knowledge storage, connecting atom identities to specific entity clusters as the case studies suggest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces 'Atom Theory,' proposing that LLM internal representations decompose into 'atoms' defined by near-orthogonality under a non-Euclidean atomic inner product (AIP), S=(DD^T)^{-1}. It proves coherence-based RIP, uniqueness and exact l1 recoverability (Thms 8-9), and an identifiability result for threshold-activated sparse autoencoders (Thm 10). Empirically, TSAEs trained on CounterFact entity activations achieve ~99.9% R^2, learned dictionaries show low NAIP coherence, and up to 99.9% of atoms pass the proposed quantile criterion, compared with 0.5% for neurons and 68.2% for GemmaScope/LlamaScope features.

Significance. If the central claims held, the framework would be a substantial contribution: formal representational units with recovery guarantees and a practical identification method. The paper's scale of experiments and public code are assets, and the compressed-sensing arguments in Thms 8-9 are standard and appear correct. However, the main empirical validation is undermined by the circular definition of the metric used to certify atomicity, the identifiability theorem does not prove the stated equality, and the comparison against neurons/features is confounded. The theoretical results cannot, as presented, support the paper's central claim that atoms are fundamental representational units of LLMs.

major comments (4)
  1. [§4.2 / Def. 6 / Eq. (4.4)] The atomicity test is circular. The atom set D is partly defined by near-orthogonality under S=(DD^T)^{-1}, and the same S is then used to compute the NAIP coherence matrix G. For any full-rank square D, G=I exactly; for an overcomplete random D, G is a projection and its normalized off-diagonal coherence is typically O(1/sqrt(H)), matching the values in Table 1 and Figure 6. Thus the reported atomicity is a generic property of whitening a full-rank dictionary, not evidence that learned directions are special. A null model using a random D of the same shape should be reported before any claim of atomicity can be accepted.
  2. [Appendix B / §3.1] The representation-shift correction is also self-fulfilling. When S is estimated as (M M^T)^{-1} from the same activation matrix M used to produce the angle distributions, the transform whitens those samples and forces pairwise angles toward 90° in high dimensions. Figures 3 and 21-32 therefore do not reveal an underlying intrinsic geometry unless S is estimated on held-out data and the observed distribution is compared against a whitening null. The current presentation overstates the geometric discovery.
  3. [A.6 / Theorem 10] The threshold condition in the proof guarantees only support separation, not exact coefficient recovery. After thresholding, the active coordinates are z_ik = δ_ik + e_ik, and the proof shows E[e_ik]=0. But D e_i is nonzero with positive variance; expectation zero does not imply that D σ(D^T S m_i) converges to m_i in probability or almost surely. The 'probabilistic sense' is never formalized, and the additional assumption that off-diagonal G_kj are approximately N(0,s^2) is an unproved, ad hoc input. The theorem therefore does not establish the exact identifiability stated in (3.8), and §4.1's attribution of the 99.9% R^2 to Theorem 10 is not justified by the proof.
  4. [§4.3 / C.4 / C.5] The comparison against neurons and features is confounded in two ways. First, neurons and features are evaluated under the Euclidean metric while atoms are evaluated under the AIP derived from their own dictionary. Second, GemmaScope/LlamaScope features are trained on general text corpora, whereas atoms are trained on CounterFact entity activations; any q* gap may reflect training-set mismatch rather than unit quality. In addition, C.5 states that for features the quantile q is only an 'ideal upper bound' because they fail to satisfy the reconstruction equality required by Theorem 9, yet the abstract and §4.3 report 68.2% as a feature stability value. These issues must be resolved before the claimed superiority of atoms is established.
minor comments (5)
  1. [§3] Typo: 'embeded' should be 'embedded.' Some sentences are awkwardly phrased, e.g., 'representational distinguishability' and 'the centroid of the angle distribution between representations deviates significantly from 90°.'
  2. [Figure 4 / C.3] The omission of layers 30 and 31 for Llama3.1-8B is reported only in Appendix C.3; it should be stated in the main-text figure caption and in the reported averages.
  3. [Abstract / §4.3] The abstract reports q*=99.8%, while §4.3 reports per-model averages 99.74%, 99.88%, and 99.95%. These should be reconciled or the rounding should be described explicitly.
  4. [Definition 5 / Remark] The remark asserts that off-diagonal values follow N(0,s^2). This is an assumption, not a theorem; it should be labeled as such, and its role in Theorem 10 should be stated explicitly.
  5. [C.5] The quantile definitions mix continuous quantiles with the discrete integer-valued sparsity K. Explain how K_alpha is computed from integer counts and how the binary search handles the discrete nature of K.

Circularity Check

2 steps flagged

Atomicity and q* are by construction: the AIP metric is computed from the atom set it certifies, and the representation-shift 'correction' is a whitening artifact.

specific steps
  1. self definitional [Section 3.1 (Definition 1, Theorem 2), Section 3.2 (Definition 6), Section 4.2 Eq. (4.4)]
    "Definition 1: In the atom space D, the atomic inner product ⟨·,·⟩_S satisfies ⟨d_i,d_j⟩_S=0 for any pair of distinct atoms... Theorem 2: If the columns of D ... form a set of atoms ... then S=c^2(DD^⊤)^{-1}. ... Definition 6: Furthermore, for all i≠j, |⟨\tilde d_i,\tilde d_j⟩|≤ϵ, where \tilde d_i:=S^{1/2}d_i ... and \tilde S:=(DD^⊤)^{-1}."

    The atomicity test uses the very metric built from the dictionary being certified. Definition 1 defines the AIP as the inner product making distinct atoms orthogonal; Theorem 2 then solves S=(DD^⊤)^{-1} from D; Definition 6 accepts D as atoms only if its columns are near-orthogonal under this same S. The paper's own remark concedes the square case is trivial: when |D|=H, D^⊤\tilde S D=I exactly, so every full-rank square dictionary is perfectly atomic. For overcomplete D, the NAIP matrix is the projection D^⊤(DD^⊤)^{-1}D, whose off-diagonals for generic high-dimensional D are O(1/√H), matching the reported μ≈0.03–0.06. The reported atomicity is therefore a whitening artifact, not evidence that learned directions are special.

  2. fitted input called prediction [Appendix B (Representation Shifting), used in Section 3; Eq. (C.4)]
    "To address it, we estimate an appropriate inner product as prescribed by Theorem 2 and Corollary 3 using 100,000 Wikipedia activation samples (Meng et al., 2022). Recomputing the angle distribution yields a centroid concentrated around 90°, which accords closely with the theoretical expectation."

    The inner product S=(MM^⊤)^{-1} is estimated from the same activation distribution whose angle distribution is then reported as validation of the AIP. Applying S^{1/2} is Mahalanobis whitening: the transformed sample covariance becomes the identity (or a rank-H projection), which mathematically forces the mean pairwise angle toward 90° in high dimension. Thus the 'correction' of representation shifting is a generic property of whitening any full-rank dataset, not a discovered geometry of LLMs; the agreement with the 'theoretical expectation' is a self-consistency check of the fitted transform, not a prediction.

full rationale

The central q* stability/atomicity result is substantially circular. Definition 1 + Theorem 2 define the metric S=(DD^⊤)^{-1} from the same D that Definition 6 then certifies as atoms, so the atomicity criterion is self-referential; for square D it is an identity, and for overcomplete D the off-diagonal coherence is a generic whitened-frame property. Similarly, Appendix B's demonstration that AIP moves angle centroids to 90° is a mathematical consequence of estimating S from the measured activations, not an empirical discovery. The R^2=99.9% sparse reconstruction result is a genuine independent measurement, and the case studies are descriptive, which prevents a score of 10; however, the headline claim of 'ideal atoms statistically' rests on the by-construction q* and atomicity criteria, so the central claim is largely forced by the definitional setup. The atoms-vs-features comparison is additionally confounded by different training distributions (CounterFact entity activations vs. general-corpus SAEs) and by the paper's own caveat that the features' q* is only an upper bound because features do not reconstruct; this is a measurement concern beyond the main circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on a set of mutually defining objects: atoms are defined as dictionaries that are near-orthogonal under a metric built from the same dictionary. This creates a circularity burden. The empirical numbers also depend on free hyperparameters (SAE capacity, thresholds, sparsity penalty) and on distributional assumptions in Theorem 10 that are not verified.

free parameters (4)
  • SAE hidden size (capacity) = 16k/65k for Gemma2, 8x/32x for Llama3.1 (main results); scaling experiments vary capacity
    The SAE width is chosen by the authors; the paper's own scaling experiment (Fig. 8) shows SAE size controls reconstruction, so the near-perfect R^2 depends on having enough capacity for the data scale.
  • Threshold initialization tau = 0.001 or 0.0001
    Used in JumpReLU activation; authors state 0.001 provides a good default and is a tuned hyperparameter (App. C.3).
  • Sparsity coefficient lambda = 0.1 (main), with 0.01/1 robustness checks
    Fixed, but shown insensitive; part of the objective (Eq. 4.1).
  • Quantile level q in the uniqueness test = binary search up to 0.999999
    The reported q* statistic summarizes the fraction of samples meeting the sufficient condition; it is computed from the fitted dictionary and codes.
axioms (4)
  • domain assumption Assumption 4 (Sparsity): every representation is K-sparse in the atom dictionary with K much less than |D|
    Central assumption that the representations of interest admit a sparse code; introduced in Assumption 4 and used in all theorems.
  • standard math Softmax invariance gives a global invertible linear ambiguity in representations (Eq. 3.1)
    Used to justify the need for a non-Euclidean inner product; the reparameterization is standard.
  • ad hoc to paper The off-diagonal entries G_{kj} are approximately N(0,s^2) with mean zero, invoked in the proof of Theorem 10
    No derivation is given; the theorem's 'probabilistic sense' conclusion relies on this distributional assumption.
  • domain assumption A threshold exists satisfying delta_min > epsilon(2K-1)delta_max (support separation)
    Required for Theorem 10; not verified over trained SAE coefficients in the main text.
invented entities (2)
  • Atoms no independent evidence
    purpose: Postulated fundamental units of LLM representations, defined as sparse near-orthogonal dictionary elements under the AIP.
    The only evidence is internal: R^2 and q* computed from SAEs trained and evaluated on the same data. Case studies (App. C.6) are qualitative and not a falsifiable external handle.
  • Atomic inner product (AIP) no independent evidence
    purpose: Metric S=(DD^T)^{-1} used to define and measure atoms.
    The AIP is defined from the atom set itself, so it cannot serve as an independent benchmark for whether the atom set is atomic.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Atoms of Large Language Models." pith.science (2026). https://pith.science/paper/ESDVNHHS

@misc{pith2026250920784,
  author       = {Pith},
  title        = {Pith review of: Towards Atoms of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESDVNHHS}},
  note         = {Machine review of arXiv:2509.20784}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The fundamental representational units (FRUs) of large language models (LLMs) remain undefined, limiting further understanding of their underlying mechanisms. In this paper, we introduce Atom Theory to systematically define, evaluate, and identify such FRUs, which we term atoms. Building on the atomic inner product (AIP), a non-Euclidean metric that captures the underlying geometry of LLM representations, we formally define atoms and propose two key criteria for ideal atoms: faithfulness ($R^2$) and stability ($q^*$). We further prove that atoms are identifiable under threshold-activated sparse autoencoders (TSAEs). Empirically, we uncover a pervasive representation shift in LLMs and demonstrate that the AIP corrects this shift to capture the underlying representational geometry. We find that two widely used units, neurons and features, fail to qualify as ideal atoms: neurons are faithful ($R^2=1$) but unstable ($q^*=0.5\%$), while features are more stable ($q^*=68.2\%$) but unfaithful ($R^2=48.8\%$). To find atoms of LLMs, leveraging atom identifiability under TSAEs, we show via large-scale experiments that reliable atom identification occurs only when the TSAE capacity matches the data scale. Guided by this insight, we identify FRUs with near-perfect faithfulness ($R^2=99.9\%$) and stability ($q^*=99.8\%$) across layers of Gemma2-2B, Gemma2-9B, and Llama3.1-8B, satisfying the criteria of ideal atoms statistically. Further analysis confirms that these atoms align with theoretical expectations and exhibit substantially higher monosemanticity. Overall, we propose and validate Atom Theory as a foundation for understanding the internal representations of LLMs. Code available at https://github.com/ChenhuiHu/towards_atoms.

Figures

Figures reproduced from arXiv: 2509.20784 by Chenhui Hu, Jun Zhao, Kang Liu, Pengfei Cao, Yubo Chen.

Figure 1
Figure 1. Figure 1: Atomization of activations and corresponding superposition. (a) Activations of LLMs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representation shifting caused by adopting the Euclidean inner product, where the cen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Correcting representation shifting by identifying and adopting the atomic inner product, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sparse reconstruction R2 scores across models. GemmaScope and LlamaScope serve as standard tools for extracting features from representations. The coefficient before × denotes the ratio of SAE hidden size to representation dimensionality. R2 values below 0 are clipped to 0. • Sparse reconstruction. Training single-layer SAEs with threshold activation on Gemma2- 2B, Gemma2-9B, and Llama3.1-8B achieves 99.9%… view at source ↗
Figure 5
Figure 5. Figure 5: Spontaneous alignment between the encoder and decoder during training on Gemma2-2B. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Maximum quantile q per layer satisfying µq < 1 2Kq−1 for each model. As demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Scaling experiments on Gemma2-2B. Finally, we investigate how varying SAE scales affect recovery performance. Specif￾ically, we train SAEs of varying sizes on Gemma2-2B, gradually increasing capacity to accommodate increasing dataset size. The re￾sults in [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Representation shifting of GPT2-Small. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 30 Layer 31 Layer 32 Layer 33 Layer 34 Layer 35 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Representation shifting of GPT2-Large. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 22 Layer 23 Layer 24 Layer 25 Layer 26 Layer 27 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Representation shifting of GPT-J-6B. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Representation shifting of Pythia-1B. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 26 Layer 27 Layer 28 Layer 29 Layer 30 Layer 31 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representation shifting of Pythia-2.8B. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 26 Layer 27 Layer 28 Layer 29 Layer 30 Layer 31 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Representation shifting of Pythia-6.9B. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Representation shifting of Llama2-7B. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 34 Layer 35 Layer 36 Layer 37 Layer 38 Layer 39 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Representation shifting of Llama2-13B. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Representation shifting of Llama3-8B. Layer 0 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8 Layer 9 Layer 10 Layer 11 ... ... Layer 26 Layer 27 Layer 28 Layer 29 Layer 30 Layer 31 [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Representation shifting of Llama3.1-8B. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Representation shifting of Gemma2-2B. ... ... ... ... Layer 0 Layer 1 Layer 2 Layer 37 Layer 41 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 21
Figure 21. Figure 21: Correcting representation shifting on GPT2-Small. [PITH_FULL_IMAGE:figures/full_fig_p022_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Correcting representation shifting on GPT2-Large. [PITH_FULL_IMAGE:figures/full_fig_p023_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Correcting representation shifting on GPT-J-6B. [PITH_FULL_IMAGE:figures/full_fig_p023_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Correcting representation shifting on Pythia-1B. [PITH_FULL_IMAGE:figures/full_fig_p023_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Correcting representation shifting on Pythia-2.8B. [PITH_FULL_IMAGE:figures/full_fig_p024_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Correcting representation shifting on Pythia-6.9B. [PITH_FULL_IMAGE:figures/full_fig_p024_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Correcting representation shifting on Llama2-7B. [PITH_FULL_IMAGE:figures/full_fig_p025_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Correcting representation shifting on Llama2-13B. [PITH_FULL_IMAGE:figures/full_fig_p025_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Correcting representation shifting on Llama3-8B. [PITH_FULL_IMAGE:figures/full_fig_p026_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Correcting representation shifting on Llama3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p026_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Correcting representation shifting on Gemma2-2B. [PITH_FULL_IMAGE:figures/full_fig_p027_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Correcting representation shifting on Gemma2-9B. [PITH_FULL_IMAGE:figures/full_fig_p027_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Superposition of activations on Gemma2-2B. [PITH_FULL_IMAGE:figures/full_fig_p028_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Solving superposition on Gemma2-2B. (a) Gemma2-2B. (b) Gemma2-9B. Cumulative Samples Cumulative Samples N o r m a liz e d R a n k N o r m a liz e d R a n k [PITH_FULL_IMAGE:figures/full_fig_p028_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Cumulative normalized rank of (a) Gemma2-2B and (b) Gemma2-9B. Each data point [PITH_FULL_IMAGE:figures/full_fig_p028_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Pareto front during training on Gemma2-2B. [PITH_FULL_IMAGE:figures/full_fig_p030_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Training loss is robust to hyperparameter selection on [PITH_FULL_IMAGE:figures/full_fig_p030_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: Average ℓ0 norm after sparse reconstruction cross models. where ℓ 2 is the Hilbert space of square–summable sequences, ensuring the completeness required by kernel methods. Intuitively, ϕ(x) represents the coordinates of x in the “atoms” basis. This leads to the kernel k(x,y) = ⟨ϕ(x), ϕ(y)⟩ℓ 2 (C.6) = ⟨D⊤Sx˜, D⊤Sy˜ ⟩ℓ 2 (C.7) = (Sx˜ ) ⊤DD⊤(Sy˜ ) (C.8) = x ⊤Sy˜ = ⟨x,y⟩S˜. (C.9) Thus taking the Euclidean in… view at source ↗
Figure 39
Figure 39. Figure 39: Spontaneous alignment between the encoder and decoder during training on Gemma2- [PITH_FULL_IMAGE:figures/full_fig_p034_39.png] view at source ↗
Figure 40
Figure 40. Figure 40: Spontaneous alignment between the encoder and decoder during training on Llama3.1- [PITH_FULL_IMAGE:figures/full_fig_p034_40.png] view at source ↗
Figure 41
Figure 41. Figure 41: NAIP distribution of atoms across all layers of the Gemma2-2B. [PITH_FULL_IMAGE:figures/full_fig_p034_41.png] view at source ↗
Figure 42
Figure 42. Figure 42: NAIP distribution of atoms across all layers of the Gemma2-9B. [PITH_FULL_IMAGE:figures/full_fig_p035_42.png] view at source ↗
Figure 43
Figure 43. Figure 43: NAIP distribution of atoms across all layers of the Llama3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p035_43.png] view at source ↗
Figure 44
Figure 44. Figure 44: Average ℓ0 norm of scaling experiments on Gemma2-2B [PITH_FULL_IMAGE:figures/full_fig_p036_44.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 18 linked inside Pith

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Language models can explain neurons in language models

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

    2023

  3. [3]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Learning multi-level features with matryoshka sparse autoencoders

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503.17547, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information

    Emmanuel J Cand \`e s, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory, 52 0 (2): 0 489--509, 2006

    2006

  7. [7]

    Feature hedging: Correlated features break narrow sparse autoencoders

    David Chanin, Tom \'a s Dulka, and Adri \`a Garriga-Alonso. Feature hedging: Correlated features break narrow sparse autoencoders. arXiv preprint arXiv:2505.11756, 2025

    arXiv 2025

  8. [8]

    The knowledge microscope: Features as better analytical lenses than neurons

    Yuheng Chen, Pengfei Cao, Kang Liu, and Jun Zhao. The knowledge microscope: Features as better analytical lenses than neurons. arXiv preprint arXiv:2502.12483, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Compressed sensing

    David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52 0 (4): 0 1289--1306, 2006

    2006

  11. [11]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024

    2024

  12. [12]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1 0 (1): 0 12, 2021

    2021

  13. [13]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    Pith/arXiv arXiv 2022

  14. [14]

    Decomposing the dark matter of sparse autoencoders

    Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.14670, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    Pith/arXiv arXiv 2012

  17. [17]

    Finding neurons in a haystack: Case studies with sparse probing

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    A structural probe for finding syntax in word representations

    John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.\ 4129--4138, 2019

    2019

  20. [20]

    Knowledge in superposition: Unveiling the failures of lifelong knowledge editing for large language models

    Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Knowledge in superposition: Unveiling the failures of lifelong knowledge editing for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 24086--24094, 2025

    2025

  21. [21]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35: 0 17359--17372, 2022

    2022

  23. [23]

    Emergent linear representations in world models of self-supervised sequence models

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Feature visualization

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi:10.23915/distill.00007. https://distill.pub/2017/feature-visualization

    doi:10.23915/distill.00007 2017

  25. [25]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5 0 (3): 0 e00024--001, 2020

    2020

  26. [26]

    Compute optimal inference and provable amortisation gap in sparse autoencoders

    Charles O'Neill, Alim Gumran, and David Klindt. Compute optimal inference and provable amortisation gap in sparse autoencoders. arXiv preprint arXiv:2411.13117, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019

    Fabio Petroni, Tim Rockt \"a schel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019

    Pith/arXiv arXiv 1909

  29. [29]

    Improving dictionary learning with gated sparse autoencoders

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J \'a nos Kram \'a r, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024 a

    Pith/arXiv arXiv 2024

  30. [30]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J \'a nos Kram \'a r, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024 b

    Pith/arXiv arXiv 2024

  31. [31]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosema...

    2024

  33. [33]

    Wikidata: a free collaborative knowledgebase

    Denny Vrande c i \'c and Markus Kr \"o tzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57 0 (10): 0 78--85, 2014

    2014

  34. [34]

    Addressing feature suppression in saes

    Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes. In AI Alignment Forum, volume 6, 2024

    2024

  35. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  36. [36]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  37. [37]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  38. [38]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.