Estimating the Probability of Sampling a Trained Neural Network at Random

Adam Scherlis; Nora Belrose

REVIEW 4 major objections 5 minor 27 references

Estimating the Probability of Sampling a Trained Neural Network at Random

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper estimates the 'local volume' of a trained network—the probability that random initialization lands in a region that behaves like the anchor—and shows the volume is astronomically small, shrinks during training, and is smaller…

desk verdict A sound, useful estimator for star-domain volumes; the presentation overstates the target quantity, but the core math and experiments deserve serious review. read the letter →

arxiv 2501.18812 v2 pith:2RJHDDFA submitted 2025-01-31 cs.LG

Adam Scherlis , Nora Belrose This is my paper · ORCID

classification cs.LG MSC 68T0762B1065C05

keywords localvolumestardomainKLdivergenceneuralnetworkcomplexityhypothesisimportancesamplingminimumdescriptionlengthgeneralization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the 'local volume' of a trained neural network—the probability that a fresh random initialization would land in a region of parameter space where the network behaves essentially like the anchor network—can be estimated quickly and used as a practical complexity measure. It adapts an existing basin-volume estimator and corrects its main weakness: naive Monte Carlo is unbiased but severely underestimates the volume, because the log-volume is dominated by rare long directions. The correction is an importance-sampling preconditioner that stretches the sampling along flat directions, using Hessian or optimizer-buffer information that is already available, while Markov's inequality turns the naive estimate into a very confident lower bound. The paper argues that negative log local volume behaves like an information-content measure: it grows during language-model training and is larger for poisoned, overfit networks that generalize badly. The headline numerical example is roughly one in $10^{3.6\times 10^8}$ for a 31M-parameter language model.

What carries the argument

The central object is the largest star domain $S$ anchored at a parameter vector $\theta$ such that a cost function $C$ (here, expected KL divergence from the anchor over a fixed input set) stays below a threshold $\epsilon$. Because a star domain is fully determined by its radial function $r(u)$, the distance from the anchor to the boundary along each unit direction $u$, its volume reduces to $\mathrm{vol}(S) = \frac{|S^{N-1}|}{N}\,\mathbb{E}_u[r(u)^N]$; the estimator samples $k$ directions and replaces the expectation with a sample mean. The key improvement is a positive-definite preconditioner $P$ with unit determinant, which rotates and stretches the sampled directions before normalizing them; the importance-sampling weight $|v|^N$ in Eq. (8) keeps the estimator unbiased while directing samples toward the long, flat directions that dominate the volume. For the Gaussian-initialization case, the radial integral is replaced by a one-dimensional integral of the Gaussian density, evaluated with a Laplace-type second-order expansion. The paper uses the Hessian, its diagonal, the HesScale approximation, or Adam's second-moment buffers as choices for $P$, and treats larger estimates as more accurate because of the Markov-inequality lower-bound guarantee.

What would settle it

On a small trained network, scan many line segments from the anchor and record the KL divergence along each one; if any segment leaves the below-cutoff set and later re-enters it, the set is not star-shaped, and the radial estimator measures a different set from the behavioral neighborhood, so the claimed lower bound no longer applies to that neighborhood.

Watch

Extended reading notes

Core claim

The central discovery is that the volume of a behaviorally-defined neighborhood around a network can be estimated from a small number of radial first-crossing measurements, and that a unit-determinant preconditioner makes the estimator far less biased in practice without changing the unbiasedness of the underlying Monte Carlo integral. Formally, the neighborhood is the largest star domain anchored at $\theta$ in which the cost, typically the expected KL divergence from $\theta$ over an input set, stays below a cutoff $\epsilon$; its volume under the Gaussian initialization measure is an expectation over unit directions of $r(u)^N$, where $r(u)$ is the distance to the boundary along direction $u$. The naive estimator of this expectation (Eq. 7) is unbiased but suffers a huge Jensen gap; the paper proves that the preconditioned estimator (Eq. 8) is still unbiased, that Markov's inequality makes overestimates beyond $\log k$ very unlikely, and that the log-volume is approximately the maximum of the log-radii in high dimension. Empirically, the estimated log-probability decreases monotonically across Pythia 31M checkpoints, is smaller for poisoned ConvNeXt at late training, and tracks a power law with slope $n/2$ at small cutoffs, consistent with a quadratic cost surface. These results are offered as evidence for the 'volume hypothesis' that architecture-driven volume disparities, not optimizer bias, determine which low-loss functions are found.

Load-bearing premise

The load-bearing premise is that the region of parameter space where the KL divergence from the anchor stays below the cutoff is star-shaped with respect to the anchor, so that radial first-crossing lengths fully determine its volume; the paper does not empirically verify star-shapedness on real networks.

Editorial extensions

If this is right

Local volume can be estimated with on the order of 100 binary-search radius computations per anchor, using Adam's existing second-moment buffers as a preconditioner, so the method scales to networks with tens of millions of parameters and costs little beyond a few hundred forward passes.
Negative log local volume supplies a minimum-description-length interpretation: a smaller neighborhood means a longer description length, so comparing two low-loss networks ranks them by complexity without needing ground-truth labels.
The $n/2$ log-log slope of volume versus cutoff at small cutoffs implies that KL neighborhoods behave like quadratic (elliptical) cost surfaces in high dimension, a behavior that contrasts with the more singular scaling seen in local-learning-coefficient analyses.
If the volume hypothesis is correct, the optimizer's role is to reach low-loss regions, while the architecture's role is to make simple functions occupy more prior measure; this would explain generalization without attributing it to the implicit bias of gradient descent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The poor performance of the full-Hessian preconditioner relative to diagonal/HesScale/Adam variants suggests that the longest axes of the KL neighborhood are not aligned with the Fisher's top eigen-directions at the anchor; a testable prediction is that a preconditioner built from the neighborhood's own radial covariance would outperform all of them.
The star-shapedness assumption could be checked directly with the same binary-search machinery: sample directions, find the first crossing, then evaluate the cost slightly beyond it; if the cost drops back below cutoff on any ray, the estimator's target set differs from the intuitive behavioral neighborhood.
Applying the estimator to random initializations before training would turn it into an architecture-only inductive-bias score; under the volume hypothesis, that score should predict which architectures are easier to train and which are more likely to generalize.
Because the estimator only needs the KL divergence on a fixed input set, it could be focused on narrow behavioral slices (e.g., math problems), making local volume a per-capability complexity measure; the paper gestures at this direction when it mentions detecting backdoors or scheming, but the per-task formulation is not developed there.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

This paper delivers a fast estimator for the local volume of a trained network's behaviorally-defined neighborhood, and it mostly delivers on its claims. The genuinely new piece is the preconditioned importance-sampling estimator in Eq. (8). Building on Huang et al.'s basin-volume method, the authors show that a unit-determinant preconditioner changes coordinates without changing volume, and the resulting estimator is unbiased. The analysis in Appendix C—Jensen gap, Markov lower bound—is correct and clearly explained. The empirical trends are coherent and worth having: local volume decreases during training, and poisoned networks show smaller volumes on held-out data. The Pythia number is a nice illustration of scale, even if it is a lower bound rather than a true probability.

What concerns me: the target quantity is not literally the probability of sampling a network within ε KL. It is the measure of the largest star domain inside that sublevel set. The authors define it that way in §2, so the estimator is internally consistent, but the title and abstract sell it as the neighborhood's size. If the true set has holes or disconnected lobes, the star envelope can be orders of magnitude smaller. They don't test star-shapedness, and I think that's a real external-validity gap.

Second, the hyperparameter ε is tuned to maximize the estimate before reporting. For a fixed ε, Markov's inequality supports a confident lower bound. After selecting the max across several ε's, that guarantee weakens. The paper should acknowledge this selection effect; currently it reads as if "larger is always more accurate" applies unconditionally.

Third, there's no code or ground truth beyond the quadratic toy model, and the binary search breaks down at low cutoffs. The implementation details matter for the lower-bound claim, and we can't check them.

The missing citations in §3.3 and §4.2 are sloppy but minor.

This is a serious paper from competent people. The math checks out, the experiments address an important question, and the tool looks genuinely useful for generalization diagnostics and MDL-based interpretability. It deserves full peer review. I'd want the authors to state explicitly that the estimates are lower bounds for the star envelope, to address the ε-selection issue, and to release code. Even without those fixes, the core method is publishable.

Referee Report

4 major / 5 minor

Summary. The paper introduces an estimator for the volume, under a Gaussian or uniform measure, of the largest star domain anchored at a given parameter vector within a sublevel set of a cost function, typically KL divergence. The estimator adapts the radial-binary-search method of Huang et al. (2020), adds an importance-sampling preconditioner, and proves unbiasedness for the star-domain volume. The authors prove that the log of the estimator is downward biased via Jensen's inequality and give a Markov-inequality lower bound. They apply the method to MLP, ConvNeXt, and Pythia models, reporting that local volume decreases during training and is smaller for poisoned or overfit networks, and they connect this to the volume hypothesis and minimum description length.

Significance. If the estimator measures the quantity claimed, it provides a computationally cheap interpretability metric that could inform generalization and complexity studies. The unbiasedness proof (Eq. 8) and the lower-bound argument (Eq. 27) are valuable theoretical contributions. The empirical results, including the extreme probability estimate for a trained language model, are striking and potentially relevant to the volume-hypothesis literature. However, the identification of the star-domain volume with the behavioral neighborhood is unverified, and the lack of ground-truth validation tempers the significance.

major comments (4)

[§2, §5.1, §7] The quantity estimated is the volume of the largest star domain contained in the sublevel set {θ': C(θ')<ε}, not the volume of the sublevel set itself. If the sublevel set has holes, disconnected lobes, or re-entrant boundaries, the star envelope is a strict subset and the estimator can be orders of magnitude smaller than the probability of sampling a network within ε KL. This identification is load-bearing for the title, abstract, and all volume-hypothesis conclusions. The manuscript does not provide any empirical check of star-shapedness on real networks, and §7 concedes that it is unclear how close the estimates are to ground truth. The authors should either test star-shapedness (e.g., by checking whether all interior points on sampled rays are within the sublevel set) or explicitly frame the claims as being about the star envelope.
[§6.1, Appendix C] The hyperparameter ε is tuned separately for each preconditioner to obtain the largest estimate. This selection invalidates the Markov-inequality guarantee of Eq. (27), which applies to a fixed estimator; after searching over ε, the probability of overestimation can be much larger than 1/k. Consequently, the statement in §5.2 that larger estimates are always more accurate is not justified for the reported numbers. The authors should either fix ε a priori based on theory or a validation set, apply a correction for multiple comparisons, or demonstrate that the qualitative results are insensitive to ε over a wide range.
[§6.3, Figure 7] The reported plummeting of poisoned ConvNeXt local volume at cutoff 10^-6 nats is identified as a floating-point issue with the binary search, yet the data point remains in the figure without any marker. This could be misread as a genuine finding. The authors should remove this point or clearly flag it as unreliable in the figure and the caption.
[§7, Appendix A] No ground-truth volumes are provided for any real network; the only calibration is the quadratic toy model in Appendix A. Since the star-envelope volume may differ substantially from the behavioral-neighborhood volume, the empirical trends (decrease during training, poisoning effect) could be artifacts of the estimator. The authors should include a small-scale experiment with an independent volume estimate (e.g., via dense Monte Carlo on a tiny MLP or SGLD thermodynamic integration) to support the claim that the estimator tracks the true volume.

minor comments (5)

[§2, Eq. (2)] There are several typos: 'neighbhorhood' appears in §2, and the sum in Eq. (2) reads 'i=i' instead of 'i=1'.
[§3.3, §4.2] There are missing references marked by question marks: a placeholder after 'often generalize worse' in §3.3 and after 'behavioral loss used in' in §4.2.
[Figures 1–3, 8] The label 'log10 prob estimate (×10^6)' is confusing because the axis values themselves are already in units of 10^6; consider rephrasing to 'log10 of probability estimate' with the multiplier indicated separately.
[§5.2] The statement 'the theoretical results above still apply' would benefit from an explicit remark that Eq. (8) remains unbiased for the Gaussian estimator in §5.3, or from a reference to a proof in the appendix.
[§6] The main text states k=100 samples per data point except where otherwise noted, but Figures 1 and 2 use k=3000 and k=1000 respectively; stating the sample count in each caption would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the estimator's unbiasedness is proven from first principles, the target quantity is defined independently of any fitted prediction, and the sole self-citation appears in a speculative future-work sentence.

full rationale

The paper's derivation chain is self-contained rather than circular. The target quantity is formally defined in Section 2 as the measure of the largest anchored star domain with cost below a cutoff, and the radial estimator in Equations 7 and 8 is an exact unbiased Monte Carlo representation of that quantity: Eq. 8 follows from a unit-determinant change of coordinates, and Eq. 27 is a direct application of Markov's inequality. No fitted parameter is renamed as a prediction: the preconditioners are used to reduce estimator variance, and the empirical trends (decreasing volume during training, smaller volumes for poisoned networks) are comparisons of independently computed estimates that are not forced by the estimator's construction; the estimator could in principle have produced opposite trends. The only self-citation (Belrose & Scherlis 2024) occurs in the future-work discussion and is not load-bearing. The paper does contain genuine limitations that are not circularity: the star-domain envelope may be a strict subset of the KL-sublevel set if that set is not star-shaped, so the estimates may not equal the intuitive probability of sampling within a given KL divergence, and the authors concede in Section 7 that 'it is still unclear how close our estimates are to the ground truth.' Preconditioner hyperparameters are tuned to maximize the estimate under a 'larger is more accurate' heuristic, which can introduce selection bias, but the naive estimator is shown alongside and the qualitative conclusions do not reduce to the tuned values. These are validity and interpretability concerns, not input-output identities, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the star-domain characterization, exact radial radii via binary search, the Gaussian integration approximation, and the heuristic that larger estimates are more accurate. Free parameters are the preconditioner epsilon and the KL cutoff. No new physical or conceptual entities are introduced; 'local volume' is a defined measurable quantity, not a postulated object.

free parameters (2)

Preconditioner epsilon = 0.1, 0.01, 0.001, 1e-5, 1e-2 (per preconditioner and cutoff setting)
Used in P ∝ 1/(|D|^{1/2}+epsilon) variants; tuned separately for Hessian, diag(H), HesScale, and Adam to maximize the volume estimate (§6.1). This choice affects all reported local volumes and comparisons.
KL cutoff epsilon = 1e-2 nats default; varied from 1e-6 to 100 nats
Defines the behavioral neighborhood; all headline results use the default cutoff. The method is designed to measure volume at a specified cutoff, but the choice is a free modeling decision.

assumptions (6)

domain assumption The low-cost behavioral neighborhood is a star domain with anchor at the center; volume equals the radial integral.
§5.1 parameterizes S by a radial function r(u); if the region has holes or disconnected lobes, first-crossing radii measure a different set.
domain assumption The radius r(u) can be found exactly enough by binary search along each ray.
§5.1 and §5.3; §6.3 reports binary search failure at 1e-6 nats, so exactness is limited by floating point.
standard math The Gaussian integral can be evaluated with a second-order Laplace-type approximation with error below floating-point precision.
§5.3; claimed but not shown in detail.
standard math A unit-determinant preconditioner preserves true volume and the estimator remains unbiased.
§5.2 change-of-coordinates argument and Eq. 8; requires P positive definite and numerical normalization.
ad hoc to paper Comparing estimates for different anchors with the same preconditioner method is meaningful; larger estimates are more accurate.
Markov bounds protect against large overestimates, but the paper also uses this heuristic to justify tuning epsilon and to compare poisoned versus unpoisoned networks.
domain assumption The initialization distribution is an appropriate prior and measure for sampling probabilities.
§4.1 and §5.3; used for the Pythia probability headline and the MDL interpretation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Estimating the Probability of Sampling a Trained Neural Network at Random." pith.science (2026). https://pith.science/paper/2RJHDDFA

@misc{pith2026250118812,
  author       = {Pith},
  title        = {Pith review of: Estimating the Probability of Sampling a Trained Neural Network at Random},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RJHDDFA}},
  note         = {Machine review of arXiv:2501.18812}
}

read the original abstract

We present and analyze an algorithm for estimating the size, under a Gaussian or uniform measure, of a localized neighborhood in neural network parameter space with behavior similar to an ``anchor'' point. We refer to this as the "local volume" of the anchor. We adapt an existing basin-volume estimator, which is very fast but in many cases only provides a lower bound. We show that this lower bound can be improved with an importance-sampling method using gradient information that is already provided by popular optimizers. The negative logarithm of local volume can also be interpreted as a measure of the anchor network's information content. As expected for a measure of complexity, this quantity increases during language model training. We find that overfit, badly-generalizing neighborhoods are smaller, indicating a more complex learned behavior. This smaller volume can also be interpreted in an MDL sense as suboptimal compression. Our results are consistent with a picture of generalization we call the "volume hypothesis": that neural net training produces good generalization primarily because the architecture gives simple functions more volume in parameter space, and the optimizer samples from the low-loss manifold in a volume-sensitive way. We believe that fast local-volume estimators are a promising practical metric of network complexity and architectural inductive bias for interpretability purposes.

Figures

Figures reproduced from arXiv: 2501.18812 by the authors.

**Figure 1.** Results (k = 3000) for various preconditioners on a small MLP. Vertical dashed lines indicate the aggregated logvolume estimate, which is very close to the maximum sample. We estimate the local volume for our small MLP, using the various preconditioners described above, as shown in 1. We show a histogram of the individual samples, with vertical dashed lines for the aggregated estimate. Note that, on a log scale, th… view at source ↗

**Figure 2.** Results (k = 1000) with and without Adam preconditioner on Pythia 31M 9.0 8.5 8.0 7.5 7.0 6.5 6.0 log10 probability estimate (×10 6 ) 0.0 0.2 0.4 0.6 0.8 1.0 ConvNeXt KL neighborhoods: preconditioning Naive Naive (poisoned) Adam Adam (poisoned) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Results (k = 1000) with and without Adam preconditioner on ConvNeXt Atto poisoned and unpoisoned (µ) preconditioner is somewhere in between. The hyperparameter ϵ is tuned separately for each of these, to obtain the largest (hence most accurate) result. We find that ϵ = 0.1 works best for the Hessian, while ϵ = 0.01 is best for diag(H) and HesScale and ϵ = 0.001 is best for both Adam preconditioners. We find it surp… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Local volume decrease while training Pythia 31M 11Note, however, that the na¨ıve estimate for the unpoisoned ConvNeXt network has a large outlier sample that completely dominates the aggregated estimate, nearly reaching the bulk of the preconditioned estimates. 7 [PIT…

**Figure 6.** Figure 6: Results for various cutoffs on Pythia 31M 10 6 10 5 10 4 10 3 10 2 10 1 10 0 10 1 10 2 cutoff (nats) 25.0 22.5 20.0 17.5 15.0 12.5 10.0 7.5 5.0 lo g10 pro b estim ate (×10 6 ) ConvNeXt KL neighborhoods: varying cutoff Naive Adam Naive (poisoned) Adam (poisoned) quadrat…

**Figure 5.** Figure 5: Local volume decrease while training ConvNeXt V2 Atto, and training metrics across datasets As expected, local volume tends to decrease during training, as the network learns more and its description length increases. For Pythia, this decrease is smooth and approxima…

**Figure 8.** Figure 8: Results (k = 1000) with and without Adam preconditioner on ConvNeXt Atto poisoned and unpoisoned 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: Local volume decrease while training ConvNeXt V2 Atto 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 7 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[3]

and Kaynak, C

Alpaydin, E. and Kaynak, C. Optical Recognition of Handwritten Digits . UCI Machine Learning Repository, 1998. DOI : https://doi.org/10.24432/C50P49

doi:10.24432/c50p49 1998
[4]

Understanding Gradient Descent through the Training Jacobian

Belrose, N. and Scherlis, A. Understanding gradient descent through the training jacobian, 2024. URL https://arxiv.org/abs/2412.07003

work page Pith review arXiv 2024
[5]

Benton, G., Maddox, W., Lotfi, S., and Wilson, A. G. G. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp.\ 769--779. PMLR, 2021

work page 2021
[6]

G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

2023
[7]

Y., Bansal, A., Geiping, J., Goldblum, M., and Goldstein, T

Chiang, P.-y., Ni, R., Miller, D. Y., Bansal, A., Geiping, J., Goldblum, M., and Goldstein, T. Loss landscapes are all you need: Neural network generalization can be explained without the implicit bias of gradient descent. In The Eleventh International Conference on Learning Representations, 2022

2022
[8]

Sharp minima can generalize for deep nets

Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp.\ 1019--1028. PMLR, 2017

2017

Show all 27 references

[9]

and Mahmood, A

Elsayed, M. and Mahmood, A. R. Hesscale: Scalable computation of hessian diagonals. arXiv preprint arXiv:2210.11639, 2022

2022 arXiv
[10]

Sharpness-aware minimization for efficiently improving generalization

Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021

2021
[11]

Hinton, G. E. and Van Camp, D. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp.\ 5--13, 1993

1993
[12]

and Schmidhuber, J

Hochreiter, S. and Schmidhuber, J. Flat minima. Neural computation, 9 0 (1): 0 1--42, 1997

1997
[13]

The developmental landscape of in-context learning

Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. The developmental landscape of in-context learning. arXiv preprint arXiv:2402.02364, 2024

2024 arXiv
[14]

R., Emam, Z

Huang, W. R., Emam, Z. A. S., Goldblum, M., Fowl, L. H., Terry, J. K., Huang, F., and Goldstein, T. Understanding generalization through visualizations. In ''I Can't Believe It's Not Better!'' NeurIPS 2020 workshop, 2020. URL https://openreview.net/forum?id=pxqYT_7gToV

2020
[15]

and Hinton, G

Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

2009
[16]

The local learning coefficient: A singularity-aware complexity measure, 2024

Lau, E., Furman, Z., Wang, G., Murfet, D., and Wei, S. The local learning coefficient: A singularity-aware complexity measure, 2024. URL https://arxiv.org/abs/2308.12108

2024 arXiv
[17]

D., and Blei, D

Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference, 2018. URL https://arxiv.org/abs/1704.04289

2018 arXiv
[18]

and Grosse, R

Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

2015
[19]

Mingard, C., Valle-P \'e rez, G., Skalse, J., and Louis, A. A. Is sgd a bayesian sampler? well, almost. Journal of Machine Learning Research, 22 0 (79): 0 1--64, 2021

2021
[20]

M., Hartmann, V., and Abbasnejad, E

Teney, D., Nicolicioiu, A. M., Hartmann, V., and Abbasnejad, E. Neural redshift: Random networks are not random functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4786--4796, 2024

2024
[21]

Differentiation and specialization of attention heads via the refined local learning coefficient, 2024

Wang, G., Hoogland, J., van Wingerden, S., Furman, Z., and Murfet, D. Differentiation and specialization of attention heads via the refined local learning coefficient, 2024. URL https://arxiv.org/abs/2410.02984

2024 arXiv
[22]

Algebraic geometry and statistical learning theory, volume 25

Watanabe, S. Algebraic geometry and statistical learning theory, volume 25. Cambridge university press, 2009

2009
[23]

and Teh, Y

Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.\ 681--688. Citeseer, 2011

2011
[24]

S., and Xie, S

Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I. S., and Xie, S. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16133--16142, 2023

2023
[25]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[26]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[27]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[3] [3]

and Kaynak, C

Alpaydin, E. and Kaynak, C. Optical Recognition of Handwritten Digits . UCI Machine Learning Repository, 1998. DOI : https://doi.org/10.24432/C50P49

doi:10.24432/c50p49 1998

[4] [4]

Understanding Gradient Descent through the Training Jacobian

Belrose, N. and Scherlis, A. Understanding gradient descent through the training jacobian, 2024. URL https://arxiv.org/abs/2412.07003

work page Pith review arXiv 2024

[5] [5]

Benton, G., Maddox, W., Lotfi, S., and Wilson, A. G. G. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp.\ 769--779. PMLR, 2021

work page 2021

[6] [6]

G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

2023

[7] [7]

Y., Bansal, A., Geiping, J., Goldblum, M., and Goldstein, T

Chiang, P.-y., Ni, R., Miller, D. Y., Bansal, A., Geiping, J., Goldblum, M., and Goldstein, T. Loss landscapes are all you need: Neural network generalization can be explained without the implicit bias of gradient descent. In The Eleventh International Conference on Learning Representations, 2022

2022

[8] [8]

Sharp minima can generalize for deep nets

Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp.\ 1019--1028. PMLR, 2017

2017