Neural LoFi models deep learning as layer-wise spectral filtering that selects maximal low-degree correlations, yielding a tractable surrogate for hierarchical representation learning beyond the lazy regime.
hub
arXiv preprint arXiv:2602.07488 , year =
15 Pith papers cite this work. Polarity classification is still indexing.
15
Pith papers citing it
hub tools
citation-role summary
background 2
citation-polarity summary
years
2026 15roles
background 2polarities
background 2representative citing papers
Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.
The Random Language Model exhibits a hierarchy of phase transitions in the double-scaling limit ε̃_d → 0, N → ∞ at fixed x = ε̃_d log N, with symbol correlations, non-uniform marginals, and glassy freezing, yielding scaling laws consistent with large language models.
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
Critical percolation clusters embedded in high dimensions, combined with taxonomic latent variables, form an analytically tractable synthetic data model whose ground-truth hierarchy can be linearly decoded from network activations.
Transformer wave functions for the J1-J2 Heisenberg model exhibit size-independent power-law decay of V-score with compute, with the exponent decreasing as frustration increases.
Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B parameters.
Uniform-based discrete diffusion models behave as associative memories that retrieve unseen data, with a dataset-size-driven memorization-to-generalization transition detectable via conditional entropy of token predictions.
Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.
Rosetta Neurons in language models up to 30B and vision models up to 5B parameters scale sublinearly with size while becoming more selective and monosemantic.
U-turn chains are Markov chains formed by short forward-backward diffusion steps that remain on the learned manifold and, with Metropolis-Hastings, sample from energy-modified targets, exhibiting an ergodicity-breaking transition on fragmented manifolds.
A sparse-activation model predicts double-descent loss with distinct under- and over-parameterized scaling exponents set by sparsity, plus a compute-optimal frontier favoring dataset growth.
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
A mechanics of the learning process is emerging in deep learning theory, characterized by dynamics, coarse statistics, and falsifiable predictions across idealized settings, limits, laws, hyperparameters, and universal behaviors.
Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.
citing papers explorer
-
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
Neural LoFi models deep learning as layer-wise spectral filtering that selects maximal low-degree correlations, yielding a tractable surrogate for hierarchical representation learning beyond the lazy regime.
-
Smooth Scaling Laws Hide Stepwise Token Learning
Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.
-
Phase structure of the Random Language Model
The Random Language Model exhibits a hierarchy of phase transitions in the double-scaling limit ε̃_d → 0, N → ∞ at fixed x = ε̃_d log N, with symbol correlations, non-uniform marginals, and glassy freezing, yielding scaling laws consistent with large language models.
-
Learn from your own latents and not from tokens: A sample-complexity theory
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
-
Critical Percolation as a Synthetic Data Model for Interpretability
Critical percolation clusters embedded in high dimensions, combined with taxonomic latent variables, form an analytically tractable synthetic data model whose ground-truth hierarchy can be linearly decoded from network activations.
-
Scaling Laws for Neural-Network Quantum States
Transformer wave functions for the J1-J2 Heisenberg model exhibit size-independent power-law decay of V-score with compute, with the exponent decreasing as frustration increases.
-
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B parameters.
-
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Uniform-based discrete diffusion models behave as associative memories that retrieve unseen data, with a dataset-size-driven memorization-to-generalization transition detectable via conditional entropy of token predictions.
-
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.
-
Neuron Populations Exhibit Divergent Selectivity with Scale
Rosetta Neurons in language models up to 30B and vision models up to 5B parameters scale sublinearly with size while becoming more selective and monosemantic.
-
Sampling Data with Chains of Forward-Backward Diffusion Steps
U-turn chains are Markov chains formed by short forward-backward diffusion steps that remain on the learned manifold and, with Metropolis-Hastings, sample from energy-modified targets, exhibiting an ergodicity-breaking transition on fragmented manifolds.
-
Asymmetric Scaling Laws from Sparse Features
A sparse-activation model predicts double-descent loss with distinct under- and over-parameterized scaling exponents set by sparsity, plus a compute-optimal frontier favoring dataset growth.
-
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
-
There Will Be a Scientific Theory of Deep Learning
A mechanics of the learning process is emerging in deep learning theory, characterized by dynamics, coarse statistics, and falsifiable predictions across idealized settings, limits, laws, hyperparameters, and universal behaviors.
-
Statistical Properties of Training & Generalization
Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.