REVIEW 4 major objections 5 minor 24 references
Representation Learning on a Random Lattice
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper claims that the geometry of a generic data distribution—modeled as a random lattice—forces component features in neural networks to be four-dimensional, because percolation clusters have fractal dimension 4.
desk verdict A clear and honest conceptual paper that borrows percolation theory from the author's prior work, but its most concrete claim—that component features need four-dimensional representations—is an analogy, not a derivation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the argument is site percolation on a $d$-dimensional hypercubic lattice, analyzed through the exactly solvable Bethe-lattice approximation for $d\ge 6$. The load-bearing outputs are the cluster size distribution $n_s \propto s^{-5/2}e^{-cs}$ and the fractal dimension $D=4$, derived by equating the correlation-length exponents in chemical and Euclidean distance using the random-walk relation $r^2\propto l$. These cluster statistics are then read as feature statistics: cluster identity becomes context features, within-cluster coordinates become component features, and departures from ideal general-purpose learning become surface features.
What would settle it
Find a well-isolated component feature in a trained transformer—one that tracks a compositional relation within a cluster—and measure the intrinsic dimension of its activation subspace; the paper predicts $4$, so a stable intrinsic dimension clearly different from $4$ (for example, the 2-dimensional circular features already reported for days of the week) would refute the central prediction.
Extended reading notes
Core claim
The paper's central claim is that a generic data distribution, viewed through the lens of a general-purpose learner, has the geometry of a random percolation cluster, and that this geometry fixes the structure of the features a network must learn. At or near the percolation threshold in high dimension ($d\ge 6$), finite clusters are treelike, have a power-law size distribution with exponent $\tau=5/2$, and have fractal dimension $D=4$. From this the paper concludes that context features identify clusters and their fractal substructure, component features provide coordinates inside a cluster, and component features should generically require four-dimensional representations. This is presented as a quantitative, architecture-independent alternative to the linear representation hypothesis, qualified as consistent with, rather than proven by, existing mechanistic interpretability findings.
Load-bearing premise
The load-bearing premise is that the fractal dimension of percolation clusters in input space ($D=4$) directly forces the number of coordinates a network must use to represent a component feature to be four; the paper does not show a mechanism by which data-space cluster geometry constrains the coding dimension of activations.
Editorial extensions
If this is right
- Sparse dictionary learning should be most effective for datasets in the subcritical regime; near and above the percolation threshold, features should increasingly activate in dense composition rather than sparse superposition.
- Component features should be multidimensional rather than one-dimensional, so interpretability tools that search for 1D linear features will miss part of the structure.
- Feature splitting and absorption seen in larger sparse autoencoders reflect an underlying nested hierarchy of context features, not merely a failure of dictionary learning.
- The relative numbers of context and component features shift with the occupation regime: many of both below threshold, few of both just above, and only component features when the infinite cluster becomes Euclidean.
Reading between the lines
- The author leaves implicit that the $D=4$ prediction is testable without knowing the full data distribution: one can measure the intrinsic dimensionality of component-feature subspaces directly in trained models, and the predicted universality across architectures and modalities would be a strong signal.
- A natural extension is to replace hard bonds with weighted bonds, turning crisp percolation clusters into graded family-resemblance concepts; this suggests testable predictions about how feature hierarchies soften as cluster membership becomes graded.
- If the random-input-format assumption fails for modalities with strong built-in symmetries, the 4D prediction might hold for high-level semantic features while low-level features track the modality's own geometry, implying that tests should stratify by layer or feature type.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes modeling a generic data distribution as a site-percolation model on a d-dimensional lattice. It reviews percolation theory (Appendix A) to obtain cluster size distribution exponent tau=5/2 and fractal dimension D=4 for d>=6. It classifies learned features into context, component, and surface features, and argues that this taxonomy and the percolation geometry explain observations from mechanistic interpretability, including feature sparsity, hierarchy, feature splitting, and multidimensional features. It predicts that component features generically require four-dimensional representations, and that feature usage has a power-law distribution.
Significance. If the proposed link between percolation geometry and learned representations were established, this would provide a unifying geometrical foundation for feature decompositions in neural networks, connecting data distributions to representation structure. The paper brings a standard statistical-physics tool into mechanistic interpretability and makes one concrete, falsifiable prediction (four-dimensional component features). Its strengths include a correct and clearly presented review of high-dimensional percolation exponents in Appendix A, and a taxonomy that plausibly organizes several existing interpretability findings. However, the central inference from percolation clusters to neural features is asserted rather than derived, and the most quantitative prediction rests on an analogy between input-space fractal dimension and activation-space dimensionality. The paper therefore reads as a promising framework or position paper rather than an established theory.
major comments (4)
- [Section 3.4, with Appendix A (Eqs. 12, 18)] The claim that component features "should generically require four-dimensional representations, as they map out treelike data volumes that have four-dimensional fractal geometry" does not follow from the derived D=4. D is the Euclidean fractal dimension of a percolation cluster embedded in the input lattice; the dimensionality of a neural representation is a property of the network's coding scheme, and no equation or mechanism connects the two. A concrete counterexample: a one-dimensional random-walk path embedded in R^6 has Euclidean fractal dimension 2 but can be encoded with a single arc-length coordinate, showing that fractal dimension of an embedding does not constrain coding dimension. Unless a mechanism is supplied (e.g., a proof that computing a cluster's target function requires a representation whose dimension equals the cluster's fractal dimension), this most concrete quantitative prediction is an analogy, not a consequence of the model.
- [Sections 2.1-3.3] The mapping from percolation clusters to features is not formalized. The model defines clusters via a bond rule on the target function (Section 2.1) and then asserts that each cluster supports a distinct target function and that context/component/surface features are "a minimal set of sparse or composable latent variables" (Section 3.1). No equations connect cluster statistics (n_s, D) to feature count, feature dimensionality, or feature activation frequency. For example, the statement in Section 3.4 that "both context and component features should have a power-law distribution in use frequency" is not derived from n_s ~ s^{-tau}; it is not even specified whether feature frequency is proportional to cluster size, cluster volume, or some other observable. The central chain from data geometry to feature statistics is therefore underived.
- [Section 3.1 and Table 1] The three-category taxonomy appears to be introduced to accommodate existing interpretability findings (Gurnee et al., Bricken et al., Engels et al., Ilyas et al.) rather than derived from the percolation model. Because the model has unconstrained parameters (p, d) and no independent criterion for which regime applies, its qualitative predictions can accommodate both sparse and dense features, and both many and few features, by tuning p. For instance, the classification of modular addition as a "surface feature" in Section 3.1 is purely verbal and not tied to any property of the percolation model. Without a calibration of p to measurable dataset characteristics (which the paper itself notes is needed in the footnote to Table 1), these are not falsifiable predictions.
- [Section 2.1] The model assumes both that the data distribution is near the percolation threshold (p ≈ pc) and that the occupied-site process is independent site percolation. These are substantive assumptions, not consequences of the target-function bond rule. All the paper's quantitative claims (power-law cluster size distribution, D=4) hold only at criticality in the scaling limit, yet no argument is given for why a generic data distribution should be critical. The paper should either derive criticality from a plausible mechanism (e.g., a maximum-entropy or self-organized criticality argument) or explicitly present it as a falsifiable assumption with observable consequences, rather than treating it as the default case.
minor comments (5)
- [Section 3.4] The sentence contrasting the predicted four-dimensional component features with the "circular feature manifolds" of Engels et al. (2024a) is confusing: the paper claims to be "in line with" Engels et al. while simultaneously predicting a different geometry. Clarify how the two claims relate.
- [Section 3.3] The sentence "This sparsity suggests that models may efficiently represent these features polysemantically, in superposition Elhage et al. (2022)" is missing a comma before the citation, making it read as if superposition is the name of a paper rather than a concept.
- [Appendix A, Eq. 8] The notation sigma=1/2 is introduced in Eq. 8 and used in later exponents; defining sigma as a standard percolation exponent at its first mention would help readers not familiar with the notation.
- [References] Several references are to blog posts and LessWrong posts (Nabeshima 2024, Bussmann et al. 2024, Mendel 2024, Yudkowsky 2008). These are appropriate for a fast-moving field, but the journal's reference guidelines should be checked for consistency.
- [Section 1] The paper criticizes Elhage et al.'s feature definition as circular, but its own definition of a feature ("coordinates for computing the target function") is not stated formally. A precise definition of 'feature' would strengthen the claims.
Circularity Check
No circularity: the model is assumed and derived via percolation theory, and the D=4-to-4D-representation claim is an analogy, not a construction-level reduction.
full rationale
The paper's derivation chain is not circular. Section 2.1 adopts, with explicitly stated assumptions, a random-lattice model whose percolation analysis is carried out independently in Appendix A using standard percolation theory (Stauffer & Aharony, 1994), rather than by appealing to the conclusions of Brill (2024); the self-citation supplies the modeling idea, but the present paper re-derives the cluster-size exponent tau = 5/2 and fractal dimension D = 4. The feature taxonomy in Section 3.1 is stipulated as a definition, not fitted to data; Table 1 is a heuristic summary, and no parameter is estimated from downstream measurements and then renamed a prediction. The one concrete prediction—that component features should generically require four-dimensional representations because percolation clusters have D = 4—rests on a substantive and contestable analogy between input-space fractal geometry and activation-space coding dimension, but it is not equivalent by construction to its premise: D = 4 is a property of percolation clusters in the input lattice, while the predicted dimension is a property of learned representations. If the analogy fails, the claim is unsupported, but that is a correctness and falsifiability concern, not circularity. No circular step can be exhibited, so the score is 0.
Assumptions & free parameters
free parameters (2)
- Occupation probability p
- Lattice dimension d
assumptions (6)
- domain assumption The data space can be modeled as a d-dimensional hypercubic lattice with sites occupied independently with probability p.
- domain assumption Target functions are context-dependent and can be decomposed into a higher-order function yielding local first-order functions; bonds connect inputs where the target function behaves the same.
- domain assumption General-purpose learning implies the input format is statistically unrelated to latent structure, so occupancy can be treated as random.
- standard math High-dimensional site percolation is well approximated by the Bethe lattice.
- ad hoc to paper Each percolation cluster supports a distinct target function, and features are the coordinates for computing that function.
- ad hoc to paper Cluster fractal dimension D=4 implies component features require four-dimensional representations.
invented entities (1)
-
Context, component, and surface feature taxonomy
Cite this review
Pith. "Pith review of Representation Learning on a Random Lattice." pith.science (2026). https://pith.science/paper/IKJQSSTH
@misc{pith2026250420197,
author = {Pith},
title = {Pith review of: Representation Learning on a Random Lattice},
year = {2026},
howpublished = {\url{https://pith.science/paper/IKJQSSTH}},
note = {Machine review of arXiv:2504.20197}
}
read the original abstract
Decomposing a deep neural network's learned representations into interpretable features could greatly enhance its safety and reliability. To better understand features, we adopt a geometric perspective, viewing them as a learned coordinate system for mapping an embedded data distribution. We motivate a model of a generic data distribution as a random lattice and analyze its properties using percolation theory. Learned features are categorized into context, component, and surface features. The model is qualitatively consistent with recent findings in mechanistic interpretability and suggests directions for future research.
Figures
Reference graph
Works this paper leans on
-
[1]
12 ξ2 l = pc(p +pc) (pc−p)2 ∝ (pc−p)−2, p<p c
= 1 +pc, yielding 6If branching continues only for a fixed number of iterations, the resulting finite structure, called a Cayley tree, has nonnegligible boundary conditions (Ostilli, 2012). 12 ξ2 l = pc(p +pc) (pc−p)2 ∝ (pc−p)−2, p<p c. (4) We can also straightforwardly compute the mean size S of finite clusters by summing over the expected number of conn...
work page 2012
-
[2]
arXiv preprint arXiv:2408.05147 ,
-
[7]
A. Demski and S. Garrabrant. Embedded agency. arXiv preprint arXiv:1902.09469 ,
arXiv 1902
-
[9]
https://distill.pub/2019/advex-bugs-discussion/response-1
doi: 10.23915/distill.00019.1. https://distill.pub/2019/advex-bugs-discussion/response-1. 9 W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610 ,
arXiv 2019
-
[10]
S. S. R. Hindupur, E. S. Lubana, T. Fel, and D. Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822 ,
- [12]
-
[13]
Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62,
work page 2022
-
[16]
Accessed: 2024-10-30. E. Michaud, Z. Liu, U. Girit, and M. Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems , 36,
work page 2024
Show all 24 references
-
[17]
Accessed: 2025-03-12. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 ,
2025 arXiv
-
[18]
html#linear-representations
URL https://transformer-circuits.pub/2024/july-update/index. html#linear-representations. M. Ostilli. Cayley trees and bethe lattices: A concise analysis for mathematicians and physicists. Physica A: Statistical Mechanics and its Applications , 391(12):3417–3423,
2024
-
[20]
C. R. Shalizi and C. Moore. What is a macrostate? subjective observations and objective dynamics. arXiv preprint cond-mat/0303625,
-
[21]
URL https://transformer-circuits.pub/2024/scaling-monosemanticity/ index.html. A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30,
2024
-
[23]
We derive properties of the clusters’ size distribution, including the power-law exponent and exponential decay away from criticality, and their fractal dimension
11 A Review of Percolation Cluster Properties This section reviews selected properties of percolation clusters, applicable when d≥ 6, s≫ 1, and p≈pc. We derive properties of the clusters’ size distribution, including the power-law exponent and exponential decay away from criti...
1994
-
[1992]
Chanin, J
D. Chanin, J. Wilken-Smith, T. Dulka, H. Bhatnagar, and J. Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507 ,
-
[1998]
Y. Li, E. J. Michaud, D. D. Baek, J. Engels, X. Sun, and M. Tegmark. The geometry of concepts: Sparse autoencoder feature structure. arXiv preprint arXiv:2410.19750 ,
-
[2008]
Accessed: 2024-03-08. Z. Zhong, Z. Liu, M. Tegmark, and J. Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems , 36,
2024
-
[2012]
K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658 ,
-
[2013]
Bereska and E
L. Bereska and E. Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082,
-
[2019]
Cunningham, A
H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 ,
-
[2020]
F. Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547 ,
1911 arXiv
-
[2022]
https://transformer-circuits.pub/2022/toy_model/index.html. J. Engels, I. Liao, E. J. Michaud, W. Gurnee, and M. Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405.14860 , 2024a. J. Engels, L. Riggs, and M. Tegmark. Decomposing the dark matter of sp...
2022 arXiv
-
[2023]
https://transformer-circuits.pub/ 2023/monosemantic-features/index.html. A. Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arXiv:2412.07942 ,
2023 arXiv
-
[2024]
Accessed: 2025-03-12. L. Cayton. Algorithms for manifold learning . eScholarship, University of California,
2025
-
[2025]
M. Huh, B. Cheung, T. Wang, and P. Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987,
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.