Pith. sign in

REVIEW 5 major objections 6 minor 4 cited by

Phase Transitions in Large Language Models and the $O(N)$ Model

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Transformer's text generation behaves as an O(N) spin model with two phase transitions — at temperature 1.2, giving the model's internal dimension, and at 7 billion parameters, marking emergent capabilities.

desk verdict The energy-temperature measurement is real; the O(N) dimension extraction is a reparameterization dressed in critical exponents. read the letter →

arxiv 2501.16241 v1 pith:S4KBDX4X submitted 2025-01-27 cs.LG cs.CLhep-thphysics.data-an

Youran Sun , Babak Haghighat This is my paper · ORCID
classification cs.LGcs.CLhep-thphysics.data-an
keywords phasetransitionsO(N)modellargelanguagemodelsscalinglawscriticalexponentsemergenceinternaldimensionTransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a Transformer's text generation can be treated as an O(N) spin model: each token is a spin and attention supplies the interactions, so the average pairwise dot product of token embeddings acts as an energy. Measuring that energy as a function of generation temperature, the authors find a phase transition at Tc ≈ 1.2, with the energy scaling like a critical system below it; the scaling exponent yields an internal dimension of roughly 5.9 to 7.3 across Qwen models of different sizes. A second, 'higher-depth' transition appears in the parameter dimension: the energy difference E(Tc) − E(∞) collapses at about 7 billion parameters, which the authors read as the emergence of a capability (awareness of generating nonsense) that small models lack. If correct, the energy-temperature curve offers a quick, test-set-free diagnostic for whether to increase model size or clean the data. The paper's central claim is that these scaling behaviors are not incidental but reflect a genuine critical phenomenon, with the same universality class as a higher-dimensional O(N) model.

What carries the argument

The load-bearing object is the mapping of a Transformer to an O(N) model: tokens are O(N) spins, attention weights define effective nearest-neighbor spin-spin couplings after weak interactions are pruned, and the energy is E = (1/$L^{2}$) Σ_{σ,τ} t_σ · t_τ. The argument runs on two scaling relations. First, near the temperature-driven transition, the specific heat is assumed to diverge as C ~ |T − Tc|^{−α}, so the energy behaves as E ~ Ec ± A|T − Tc|^{1−α}; fitting α′ gives the internal dimension via the hyperscaling identity νd = 2 − α with ν = 1/(d − 2) (Eq. (22)–(23)). Second, the parameter-driven 'higher-depth' transition is detected through the quantity E(Tc) − E(∞), plotted against parameter count without embedding, which crosses zero at Pc ≈ 7B. The paper also leans on the existence of an interacting UV fixed point of the O(N) model in 6 − ε dimensions and on the large-N exponent ν = (d − 2)^{−1} to interpret the measured exponents as those of the O(N) universality class.

What would settle it

Measure the energy-temperature curve for a model whose true internal dimension is known independently (for example, by controlling the data manifold or using a synthetic task with a known intrinsic dimension) and check whether d(α′) from Eq. (23) matches; a systematic mismatch would show that the hyperscaling conversion is not measuring the model's dimension. A cheaper test: recompute the curves under a different sampling rule (e.g., top-p or min-p instead of softmax temperature) — if the phase transition at Tc ≈ 1.2 moves or disappears, the 'criticality' is an artifact of the temperature parameterization rather than a property of the model.

Watch

Extended reading notes

Core claim

The central claim is that generated text from a Transformer is governed by the same critical physics as an O(N) model in higher dimensions. Concretely, with energy E = (1/$L^{2}$) Σ_{σ,τ} t_σ · t_τ over L tokens, the energy-temperature curve of Qwen2.5, Qwen-Math, and Qwen-Coder models shows a second-order phase transition at Tc ≈ 1.2. Fitting the scaling law E ≈ Ec ± A|T − Tc|^{1−α′} below Tc and using the hyperscaling relation νd = 2 − α with ν = 1/(d − 2) converts the fitted exponent α′ into an internal dimension d(α′) = 2(2 − α′)/(1 − α′) that ranges from 5.9 to 7.3, matching the order of magnitude of intrinsic dimension estimates. In the parameter-size direction, the paper finds that E(Tc) − E(∞), which it defines as measuring whether a model's parameter count is sufficient, vanishes at Pc ≈ 7B parameters; small models keep low energy in the nonsense phase while large models do not, a distinction the authors attribute to an emergent capability. These two transitions, the authors argue, make the Transformer an example of an RG flow from human language to machine language, with generated text as the equilibrium.

Load-bearing premise

The internal-dimension numbers depend on a hyperscaling formula that is proven only near a 6−ε-dimensional fixed point and only unitary when the spin count N exceeds 1038, yet the paper applies it to every model size without stating these conditions.

Editorial extensions

If this is right

  • The energy-temperature curve can be computed in minutes from generated text alone and would serve as a training-progress diagnostic: a drop in E after crossing Tc indicates the model needs more parameters, while a flat curve suggests focusing on data quality.
  • If the universality-class conjecture holds, the critical temperature Tc ≈ 1.2 and maximum energy Emax ≈ −4.0 should be the same for any Transformer-based model, providing a new invariant to compare architectures.
  • The transition at Pc ≈ 7B implies that increases in parameter count do not merely improve performance continuously; they trigger qualitative changes in behavior, which bears on when to expect emergent reasoning capabilities in the Qwen family.
  • The internal dimension d(α′) offers a physics-derived measure of model complexity that can be compared against intrinsic-dimension estimates, giving a cross-check on how much 'room' a model has to represent language structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same energy readout could be applied to non-Transformer architectures (SSMs, RNNs) to test whether the critical temperature and the parameter threshold are architecture-independent or specific to attention; the paper's framework does not require attention beyond defining the spin-spin coupling.
  • Because the internal dimension is extracted from a single temperature sweep, it could become a cheap proxy for comparing the representational capacity of fine-tuned versus base models, a use the authors do not explicitly explore.
  • If the hyperscaling-based interpretation is unreliable (see the load-bearing premise), the reported dimensions still stand as a two-parameter fit of the energy curve; the physically meaningful part of the claim would then reduce to the existence of the two phase transitions themselves.
  • The E(Tc) − E(∞) criterion could be tested prospectively: train a series of models at 1B, 3B, 7B, 10B on the same data and check whether the 'awareness of nonsense' behavior appears only above the fitted Pc.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes that text generated by a Transformer can be described as an O(N) model, defines an energy E in Eq. (19) as the average pairwise dot product of token representations, and measures E as a function of sampling temperature for Qwen2.5, Qwen2.5-Math, Qwen2.5-Coder, and Qwen2.5-Coder-Instruct models. It reports a phase transition at a common critical temperature Tc ≈ 1.2, fits a critical exponent α′ from the low-temperature branch, converts α′ through Eq. (23) into internal dimensions d ≈ 5.9–7.3, and identifies a second 'higher-depth' phase transition at Pc ≈ 7B parameters using E(Tc) − E(∞). The same energy is proposed as an indicator of whether a model's parameter count is sufficient for its training data.

Significance. The empirical observation that all tested Qwen models show a visually similar kink in the energy-temperature curve near T = 1.2 is interesting, and the proposed energy is cheap to compute from real model outputs; this part is falsifiable and could be useful as a descriptive diagnostic. The physical interpretation, however, is not supported. The reported internal dimension is an algebraic reparameterization of the fitted exponent α′, not an independent measurement, and the scaling relations used are applied without the conditions under which they hold. The higher-depth transition at Pc ≈ 7B is read off from the same energy curves without error bars or an independent behavioral measure, making the emergence claim circular. If the paper were reframed as an empirical study of energy-temperature curves with full statistical details and a direct test of O(N) predictions, it could make a modest contribution; in its current form the central theoretical claims are not established.

major comments (5)
  1. [§4, Eq. (23) and Table 1] The internal dimension d is not an independent output: Eq. (23) is derived by combining the hyperscaling relation νd = 2 − α with the large-N O(N) exponent ν = 1/(d − 2), so d is completely determined by the fitted α′. The manuscript does not establish that generated-text energy obeys either relation. In fact, the paper itself notes in §3 (after Eq. (14)) that the relevant 6−ε-dimensional UV fixed point is unitary only for N > 1038, while the embedding dimension N identified with the O(N) index is below this for several of the tested Qwen models, and no universality-class argument is given. The values d = 5.9–7.3 in Table 1 are therefore a reparameterization of α′, not a measurement of an internal dimension.
  2. [§4, Eq. (19)] Eq. (19) is introduced as 'the energy' without being derived from the O(N) Hamiltonian in Eq. (14). The reduction of nonlocal attention to a nearest-neighbor lattice is described only verbally, with the assertion that perturbations do not affect critical phenomena, and no explicit lattice embedding is constructed. Consequently, a power-law fit to E(T) could describe a generic softmax-temperature curve rather than a specific O(N) critical point; the claim that Tc ≈ 1.2 is a genuine critical temperature is not independently verified through, for example, susceptibility or correlation-length scaling.
  3. [§5, Table 1 and Eqs. (20)–(23)] The fit is performed on the low-temperature branch and reported as α′, but Eq. (23) is then applied as if this exponent were α. Eqs. (20)–(21) explicitly allow different exponents above and below Tc, and no argument is given for α = α′. This conflation is load-bearing because the reported internal dimension is very sensitive to the exponent: over the fitted range α′ = 0.49–0.62, d(α) changes from 5.9 to 7.3.
  4. [§5, Figure 6] The higher-depth phase transition at Pc ≈ 7B is inferred from a plot of E(Tc) − E(∞) against parameter count using only six Qwen2.5 models and no error bars, fit diagnostics, or definition of E(∞). The caption's form E ∼ log(7/P)^0.78 contains the threshold 7 as an effective fitted parameter; with three models on each side, a threshold could be placed between any adjacent pair. The interpretation that large models 'recognize' that they are generating nonsense is also circular, because 'awareness' is inferred from the same energy gap used to define the transition, with no independent behavioral test.
  5. [§5, Figure 2 and text] There is a direct inconsistency in the reported observable: the text states that the maximum energy is Emax ≈ −4.0, while Figure 2 (and Figures 3–4) plot energy values from 0 to 300. The sign and normalization of Eq. (19) are not specified, which makes the fitted values of α′ and the quantity E(Tc) − E(∞) unreproducible from the paper alone.
minor comments (6)
  1. [Figure 1 caption] The caption contains a typo: 'Demonstartion' should read 'Demonstration'.
  2. [Figures 2–4] Axis labels are inconsistent: Figure 3 uses 'T emperature' instead of 'Temperature', and the y-axis label is simply 'Energy' without indicating the normalization or sign convention.
  3. [Table 1] The column d_intrinsic cites Tulchinskii et al. (2023) but does not state which intrinsic-dimension estimator was used, the hyperparameters, or the uncertainties; this is needed to support the claim that d and d_intrinsic are 'of the same magnitude'.
  4. [§5, Experiments] The experimental description omits important details: the number of prompts or sequences, the number of seeds, whether temperature corresponds to the softmax inverse temperature β = 1/T in Eq. (7), and whether E is computed from input token embeddings or hidden states.
  5. [§5, Application] The proposed training indicator is not tested against any downstream metric, loss curve, or data-quality intervention; as written, the claim that measuring E(T) can decide whether to increase parameter size is speculative.
  6. [References] Several empirical scaling-law claims are cited to blog posts (Google, 2019; DeepSeek, 2024) rather than peer-reviewed or arXiv sources; these should be replaced or supplemented with citable references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the derivation is a fit-plus-interpretation chain with explicit conjectures, not a self-referential reduction.

full rationale

The paper's derivation chain is: define a token-token energy observable E (Eq. 19); measure E(T) for generated text; fit Eq. (21) to obtain the exponent α′; convert α′ to an internal dimension d via the external O(N) scaling relations in Eq. (22), attributed to Amit and Mati, using Eq. (23); and read off a crossover in E(Tc)−E(∞) versus parameter count, summarized by E ~ log(7/P)^0.78, as Pc ≈ 7B. None of these steps defines an input in terms of the claimed output. The α′→d conversion is indeed a one-to-one reparameterization, but that is the intended use of a scaling relation rather than a circular reduction: the paper does not use d to define E or α′, and it does compare the resulting d values against independently measured intrinsic dimensions in Table 1. The Pc ≈ 7B threshold is presented as an empirical crossover with some cross-family support from Qwen-Math and Qwen-Coder, not as a theorem-derived prediction, so the fitted zero of the displayed curve is a fit result rather than a pre-imposed conclusion. The paper explicitly labels the crucial O(N) universality-class identification as a conjecture ('We conjecture that, around the critical temperature, this dynamics is in the same universality class as the O(N) model in higher dimensions at its critical point'), and it also labels the identification of d with intrinsic dimension as a conjecture. These are validity limitations, not circularity. There are no load-bearing self-citations, and no uniqueness theorem is imported from the authors' own prior work. The main correctness risk is that Eq. (22) is imported from the 6−ε O(N) fixed point, which is unitary only for N > 1038, and the paper does not validate that LLM text lies in that universality class; if that assumption fails, the reported dimensions carry no physical content. That is an unvalidated assumption, not a circular argument, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All quantitative claims rest on fitted exponents, an arbitrary threshold, and an assumed hyperscaling relation. The theory contributes no free-parameter-free prediction beyond a reparameterization of the data.

free parameters (4)
  • critical exponent α′ per model = 0.49, 0.56, 0.58, 0.62, 0.62, 0.49 for 0.5B, 1.5B, 3B, 7B, 14B, 32B
    Fitted to the energy-temperature curve in the meaningful phase via Eq. (21); the internal dimension d(α′) is then a direct reparameterization of this fit.
  • critical temperature Tc = ≈1.2
    Read by eye from the energy-temperature curves in Figure 2, not predicted from theory.
  • higher-depth threshold Pc = ≈7B parameters
    Inferred from a six-point plot of E(Tc) - E(∞) versus parameter count, with the fitted curve E = log(7/P)^0.78 in Figure 6.
  • attention threshold = 1/√N or 1/∛N
    Arbitrarily chosen in Section 4 to construct the nearest-neighbor graph; no derivation is given for either threshold.
assumptions (4)
  • ad hoc to paper The Transformer can be approximated by a nearest-neighbor O(N) model with energy defined in Eq. (19)
    Section 4 asserts that non-local attention can be replaced by nearest-neighbor interactions with a threshold, and that critical phenomena are rigid, without a derivation.
  • domain assumption Hyperscaling relation νd = 2 - α and the large-N exponent ν = 1/(d-2) hold in the relevant regime
    Eq. (22) is taken from the large-N O(N) model in 6-epsilon dimensions and applied to fitted exponents without stating validity conditions; for d>6 or N below 1038 it is not justified.
  • ad hoc to paper The average token dot-product is the Hamiltonian of the reformulated model
    Eq. (19) defines the energy, but no derivation shows it equals the spin Hamiltonian of the O(N) model in Eq. (14).
  • domain assumption The internal dimension d equals the intrinsic dimension of the data manifold
    Section 4 conjectures that d matches d_intrinsic from Tulchinskii et al.; Table 1 shows approximate agreement but no identity is proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Phase Transitions in Large Language Models and the $O(N)$ Model." pith.science (2026). https://pith.science/paper/S4KBDX4X

@misc{pith2026250116241,
  author       = {Pith},
  title        = {Pith review of: Phase Transitions in Large Language Models and the $O(N)$ Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S4KBDX4X}},
  note         = {Machine review of arXiv:2501.16241}
}
abstract

Large language models (LLMs) exhibit unprecedentedly rich scaling behaviors. In physics, scaling behavior is closely related to phase transitions, critical phenomena, and field theory. To investigate the phase transition phenomena in LLMs, we reformulated the Transformer architecture as an $O(N)$ model. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation and the model's parameter size, respectively. The first phase transition enables us to estimate the internal dimension of the model, while the second phase transition is of \textit{higher-depth} and signals the emergence of new capabilities. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.

Figures

Figures reproduced from arXiv: 2501.16241 by the authors.

Figure 1
Figure 1. Demonstartion of how to reformulate Transformer as an O(N) model. model, as demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Energy-temperature curve of small LLMs. This figure shows the energy of Qwen2.5-0.5B. Phase transition w.r.t. temperature As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Energy of Qwen2.5 models. The temperature is used in generating text. Higher temperatures result in more random generation. The energy is computed by Eq. (19). We tested our theory on the Qwen model series (Qwen, 2024). Qwen offers the most comprehensive range of open￾source models with varying parameter sizes, from small to large: 0.5B, 1.5B, 3B, 7B, 14B, and 32B. It also of￾fers several fine-tuned versions: Qwen2.… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Energy of Qwen2.5-Math and Qwen2.5-Coder models 1 2 3 7 10 20 30 Number of Parameters without Embedding (B) 0 50 100 150 200 250 E(Tc) E( ) Qwen2.5 Qwen2.5-Math Qwen2.5-Coder Qwen2.5-Coder-Instruct E log(7/P)0.78 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Want to know whether your model’s parameter size is sufficient? Measure E(Tc) − E(∞). 2022). In our experiments, only sufficiently large models exhibit this capability. This is a novel phase transition phenomenon. When the parameter size is relatively small, the model …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Many-body Tipping Dynamics of ChatGPT-like AIs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Tipping of ChatGPT-like AI to undesirable outputs is modeled as first-passage transport of a residual-state spin across an output-basin wall, with attention disorder controlling the crossing.

  2. Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Universal adversarial suffixes can shift any text's embedding toward a model's biased mean direction, breaking embedding-based LLM safety classifiers.

  3. Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIs

    physics.soc-ph 2026-08 reject novelty 5.0 of 10

    A projection of LLM internal states, trained on some runs, predicts repetition on held-out runs and can be steered to change repetition; the headline entropy maximum is a reparameterization of an occupancy split.

  4. LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard

    cs.CL 2025-04 conditional novelty 4.0 of 10

    Fine-tuning 1.5B LLMs with SFT, DPO, and synthesized data improves scores on several Open FinLLM Leaderboard tasks, and a fitted data-scaling exponent of 0.28 is claimed to match a cross-domain value of 0.24.

Reference graph

Works this paper leans on

32 extracted references · 14 canonical work pages · cited by 4 Pith papers

  1. [1]

    Percolation in neural networks

    Alberic Torrent, J. Percolation in neural networks. 2022

    work page 2022

  2. [2]

    Field Theory, the Renormalization Group, and Critical Phenomena

    Amit, D. Field Theory, the Renormalization Group, and Critical Phenomena. International series in pure and applied physics. World Scientific, 1984. ISBN 9789971966119. URL https://books.google.at/books?id=M4yqQgAACAAJ

    work page 1984

  3. [3]

    Amit, D. J. Renormalization of the potts model. Journal of Physics A: Mathematical and General, 9 0 (9): 0 1441, sep 1976. doi:10.1088/0305-4470/9/9/006. URL https://dx.doi.org/10.1088/0305-4470/9/9/006

    work page doi:10.1088/0305-4470/9/9/006 1976

  4. [4]

    H., and Zoccolan, D

    Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. Intrinsic dimension of data representations in deep neural networks, 2019. URL https://arxiv.org/abs/1905.12784

    arXiv 2019

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  6. [6]

    Deepseek-r1-lite-preview is now live: unleashing supercharged reasoning power!, Nov 2024

    DeepSeek. Deepseek-r1-lite-preview is now live: unleashing supercharged reasoning power!, Nov 2024. URL https://api-docs.deepseek.com/news/news1120

    work page 2024

  7. [7]

    Fei, L., Giombi, S., and Klebanov, I. R. Critical O(N) models in 6- dimensions . Phys. Rev. D, 90 0 (2): 0 025018, 2014. doi:10.1103/PhysRevD.90.025018

    doi:10.1103/physrevd.90.025018 2014

  8. [8]

    Neural network flows of low q-state potts and clock models

    Giataganas, D., Huang, C.-Y., and Lin, F.-L. Neural network flows of low q-state potts and clock models. New Journal of Physics, 24 0 (4): 0 043040, April 2022. ISSN 1367-2630. doi:10.1088/1367-2630/ac63da. URL http://dx.doi.org/10.1088/1367-2630/ac63da

    work page doi:10.1088/1367-2630/ac63da 2022

Show all 32 references
  1. [9]

    Understanding searches better than ever before, Oct 2019

    Google. Understanding searches better than ever before, Oct 2019. URL https://blog.google/products/search/search-language-understanding-bert/

    2019

  2. [10]

    Building quantum field theories out of neurons, 2021

    Halverson, J. Building quantum field theories out of neurons, 2021. URL https://arxiv.org/abs/2112.04527

    2021 arXiv

  3. [11]

    Neural networks and quantum field theory

    Halverson, J., Maiti, A., and Stoner, K. Neural networks and quantum field theory. Machine Learning: Science and Technology, 2 0 (3): 0 035002, April 2021. ISSN 2632-2153. doi:10.1088/2632-2153/abeca3. URL http://dx.doi.org/10.1088/2632-2153/abeca3

    2021 doi

  4. [12]

    Conformal fields from neural networks, 2024

    Halverson, J., Naskar, J., and Tian, J. Conformal fields from neural networks, 2024. URL https://arxiv.org/abs/2409.12222

    2024

  5. [13]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017. URL https://arxiv.org/abs/1712.00409

    2017 arXiv

  6. [14]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  7. [15]

    Measuring the intrinsic dimension of objective landscapes, 2018

    Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Measuring the intrinsic dimension of objective landscapes, 2018. URL https://arxiv.org/abs/1804.08838

    2018 arXiv

  8. [16]

    S., Kawaguchi, K., Dick, R

    Lubana, E. S., Kawaguchi, K., Dick, R. P., and Tanaka, H. A percolation model of emergence: Analyzing transformers trained on a formal language, 2024. URL https://arxiv.org/abs/2408.12578

    2024 arXiv

  9. [17]

    A., and Sully, J

    Maloney, A., Roberts, D. A., and Sully, J. A solvable model of neural scaling laws, 2022. URL https://arxiv.org/abs/2210.16859

    2022 arXiv

  10. [18]

    Critical scaling in the large- N O(N) model in higher dimensions and its possible connection to quantum gravity

    Mati, P. Critical scaling in the large- N O(N) model in higher dimensions and its possible connection to quantum gravity . Phys. Rev. D, 94 0 (6): 0 065025, 2016. doi:10.1103/PhysRevD.94.065025

    2016 doi

  11. [19]

    universality class, Feb 2019

    nLab. universality class, Feb 2019. URL https://ncatlab.org/nlab/show/universality+class

    2019

  12. [20]

    Potts, R. B. Some generalized order-disorder transformations. Mathematical Proceedings of the Cambridge Philosophical Society, 48 0 (1): 0 106–109, 1952. doi:10.1017/S0305004100027419

    1952 doi

  13. [21]

    Qwen2.5: A party of foundation models, September 2024

    Qwen. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  14. [22]

    Rosenfeld, J. S. Scaling laws for deep learning. CoRR, abs/2108.07686, 2021. URL https://arxiv.org/abs/2108.07686

    2021 arXiv

  15. [23]

    and Kaplan, J

    Sharma, U. and Kaplan, J. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23 0 (9): 0 1--34, 2022. URL http://jmlr.org/papers/v23/20-1111.html

    2022

  16. [24]

    A second draft of a non-technical article on universality, Sep 2010

    Tao, T. A second draft of a non-technical article on universality, Sep 2010. URL https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-a-non-technical-article-on-universality/

    2010

  17. [25]

    Dataset: sentence-transformers/simple-wiki

    tomaarsen. Dataset: sentence-transformers/simple-wiki. https://huggingface.co/datasets/sentence-transformers/simple-wiki, 2024. Accessed: 2025-01-13

    2024

  18. [26]

    Intrinsic dimension estimation for robust detection of ai-generated texts, 2023

    Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Barannikov, S., Piontkovskaya, I., Nikolenko, S., and Burnaev, E. Intrinsic dimension estimation for robust detection of ai-generated texts, 2023. URL https://arxiv.org/abs/2306.04723

    2023 arXiv

  19. [27]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  20. [28]

    H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models, 2022. URL https://arxiv.org/abs/2206.07682

    2022 arXiv

  21. [29]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024

    Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. URL https://arxiv.org/abs/2408.00724

    2024 arXiv

  22. [30]

    Neural scaling laws from large-n field theory: Solvable model beyond the ridgeless limit, 2024

    Zhang, Z. Neural scaling laws from large-n field theory: Solvable model beyond the ridgeless limit, 2024. URL https://arxiv.org/abs/2405.19398

    2024 arXiv

  23. [31]

    Zia, R. K. P. and Wallace, D. J. Critical behaviour of the continuous n-component potts model. Journal of Physics A: Mathematical and General, 8 0 (9): 0 1495, sep 1975. doi:10.1088/0305-4470/8/9/019. URL https://dx.doi.org/10.1088/0305-4470/8/9/019

    1975 doi

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.