Dissecting a Small Artificial Neural Network

Krish Arora; Michael Bachmann; Xiguang Yang

REVIEW 3 major objections 6 minor 35 references

Dissecting a Small Artificial Neural Network

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sharp loss peaks rule a tiny network's training landscape

desk verdict Useful toy-model study; the microcanonical entropy claim needs explicit sampling domain and statistics before the barrier-free conclusion can be taken seriously. read the letter →

arxiv 2501.08341 v1 pith:BXP3EXLJ submitted 2025-01-03 cond-mat.dis-nn cond-mat.stat-mechcs.LGphysics.comp-ph

Xiguang Yang , Krish Arora , Michael Bachmann This is my paper · ORCID

classification cond-mat.dis-nncond-mat.stat-mechcs.LGphysics.comp-ph PACS 07.05.Mh

keywords XORneuralnetworklosslandscapemicrocanonicalentropydensityofbackpropagationentropicbarriersphasetransitionsinlearningfinite-sizeeffects

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes the smallest neural network that can compute the XOR logic—two input neurons, two hidden neurons, one output neuron, nine parameters in all—and asks what the geometry of its loss landscape looks like from a statistical-physics viewpoint. The central claim is that the microcanonical entropy $S(L)=\ln g(L)$, where $g(L)$ is the density of parameter states with loss $L$, is sharply peaked at the discrete losses $L=0,0.25,0.5,0.75,1$ and strongly suppressed in between. Because these peaks correspond to how many of the four training cases are answered wrongly, they act as entropic barriers that slow learning; as the hidden layer grows, the peaks weaken and the barriers disappear, so optimization in larger networks becomes essentially barrier-free. The paper also shows that weights and biases keep drifting even after the loss has converged, and that this parameter drift follows a power law whose exponent depends on network size but not on learning rate. If correct, this gives a concrete, measurable reason why larger networks train more smoothly and why learning can be viewed as an annealing process through phase-transition-like stages.

What carries the argument

The load-bearing object is the microcanonical entropy of the loss, $S(L)=\ln g(L)$ with $g(L)$ the density of parameter-space states at loss $L$. This collapses the nine-dimensional loss landscape into a one-variable curve whose peaks, slopes, and convex regions encode where learning slows down. The discrete peak locations arise from a counting argument: with a batch of four XOR cases, $k$ wrong cases contribute loss near $L=k/4$, giving peaks at $k=0,1,2,3,4$. The entropy curve is estimated by generalized-ensemble Monte Carlo sampling, and cross-sections through a reference optimal solution reveal channels, plateaus, rims, and wells at the same loss increments. The mechanism that removes the barriers is dimensionality: extra hidden neurons add parameter-space directions that let the optimizer bypass the entropic bottlenecks.

What would settle it

Recompute $S(L)$ with several independent sampling runs, explicit parameter bounds, and a convergence diagnostic; if the sharp peaks at $L=0,0.25,0.5,0.75,1$ and the intermediate suppression move or wash out when the volume or the run length is changed, the claimed entropic-barrier structure is a sampling artifact rather than a landscape property. A complementary check is to re-slice the landscape with the seven fixed parameters set to a different optimum and see whether the same channels and plateaus appear.

Watch

Extended reading notes

Core claim

Working with full batches of the four XOR cases and sigmoid activations, the authors compute the density of loss $g(L)=\int DX\,\delta(L(X)-L)$ over the nine-dimensional parameter space and define the microcanonical entropy $S(L)=\ln g(L)$. They find that $S(L)$ has pronounced peaks exactly at $L=0,0.25,0.5,0.75,1.0$—loss values at which zero, one, two, three, or all four cases in the batch are mismatched—with intermediate losses strongly suppressed. A network with one hidden neuron has no states at $L=0$, which is why a perceptron cannot represent XOR; with two or more hidden neurons, zero-loss states form a continuous valley, and the entropy curve acquires additional minor peaks that slow convergence. As $n_h$ increases from 2 to 18, these smaller entropic barriers gradually disappear, supporting the view that large networks have essentially barrier-free energy landscapes. The authors interpret convex regions of $S(L)$ as signatures of first-order-like phase transitions, so that learning is an annealing process: high-entropy, high-loss phases give way to ordered, low-entropy phases as training proceeds.

Load-bearing premise

The central claim rests on the unstated premise that the Monte Carlo density-of-loss estimate is converged, that the uniform parameter measure is the correct one for training dynamics, and that cross-sections through a single optimal solution represent the landscape.

Editorial extensions

If this is right

For the XOR network, adding hidden neurons from $n_h=2$ to $n_h=18$ smooths $S(L)$ and removes smaller peaks, implying that larger networks face weaker entropic barriers during training and converge more smoothly in the long-time regime.
Because the exponent $\gamma$ in $L(\tau)\sim\tau^{-\gamma}$ depends on the number of hidden neurons but not on the learning rate, network capacity sets the asymptotic speed of backpropagation independently of the optimizer's step size.
Weights and biases continue to drift monotonically after the loss reaches zero, so the final parameter set is not unique; any truncation of training selects one member of a continuous zero-loss channel.
Randomized-batch loss landscapes agree with nonrandomized ones after averaging over 100 scans, so mini-batch randomization preserves the main landscape features that deterministic full-batch training sees.
Convex regions of $S(L)$ can be read as phase-transition analogues, making it possible to classify learning stages (initial fluctuation, basin drop, power-law approach) as distinct thermodynamic-like phases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The discrete peak structure should generalize to any classification task with a batch of $M$ examples: entropy peaks at $L=k/M$ for $k$ mismatches, and the suppression between peaks ought to weaken as $M$ grows; measuring $S(L)$ for a small real dataset would test this directly.
If entropic barrier height can be computed from $S(L)$, it gives a quantitative criterion for the minimum hidden-layer width needed for a task to become 'easy': the smallest $n_h$ at which the minor peaks vanish, which could be checked by computing $S(L)$ at intermediate widths between 2 and 18.
The persistent parameter drift along the zero-loss channel suggests that early stopping is not merely regularization but an explicit choice among infinitely many equivalent solutions; for real data, different points on that channel may generalize differently, so entropy-based measures could be more informative than loss alone.
Because the authors connect barrier disappearance to dimensionality, a testable extension would be to initialize backpropagation at many random points in a large network and verify that the distribution of convergence times becomes unimodal and fast once $S(L)$ is convex everywhere; chaotic trajectory splitting should disappear at the same width.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a careful descriptive study of a nine-parameter XOR network, and the microcanonical entropy curves for 1–18 hidden neurons are a legitimate, mildly interesting extension of existing density-of-states work. It is not a breakthrough, and it does not need to be. The problem is that the central entropy claim is not yet supported, because Eq. (8) never specifies the integration domain or measure DX for the unbounded sigmoid parameter space. Without bounds and convergence diagnostics, Fig. 8's peaks and 'astonishing suppression' may reflect the chosen box.

What is genuinely useful: the cross-sections of the loss landscape (Fig. 6) are informative; the convergence dynamics, activation splits, parameter drift, and randomized-batch comparison are clear. The paper is honest that it is a simple case study. The phase-transition connection is framed as analogy, not derivation, which is appropriate. The self-citation to ref. [14] is not a problem; it is relevant and not load-bearing.

Soft spots, in order. (1) The density g(L) over an unbounded parameter space is formally infinite; sampling must implicitly use a finite region, but the paper never states bounds, bin width, sample counts, or convergence tests for Fig. 8. This is the load-bearing issue. The vertical shifts in Fig. 8 hide normalization, and the claim that barriers disappear with nh depends on comparing volumes across different-dimensional spaces. The stress-test note is right about this. (2) The claim that gamma is learning-rate independent rests on a few curves with no error bars; the interval [0.002, 17.484] is over-precise and should be presented as approximate. (3) The landscape cross-sections fix seven parameters at solution #1; the authors acknowledge other solutions look different but similar, so this is a minor caveat, not a fatal one. (4) The phase-transition language is more suggestive than demonstrated; the paper's own summary treats it as an analogy, so that is acceptable if claims are softened.

Verdict: the descriptive content is likely reproducible, and the entropy peaks at 0, 0.25, 0.5, 0.75, 1.0 are expected from the loss definition. But as written, the central quantitative claim lacks the essential sampling details. This is fixable with an appendix containing parameter bounds, convergence tests, and possibly error bars; the authors should also soften 'astonishing suppression' and the barrier-free conclusion.

This paper deserves a serious referee and, with revisions, could be a solid case study. I would not cite it in my own work, but I would send it out.

Referee Report

3 major / 6 minor

Summary. The paper studies a minimal 2-2-1 sigmoid network for the XOR function. It examines backpropagation convergence, the power-law decay of the loss, and two-dimensional cross-sections of the nine-dimensional loss landscape for full and randomized batches. It then introduces S(L) = ln g(L), where g(L) is the 'density of loss' estimated by Monte Carlo sampling, and uses the resulting curves to argue that the loss landscape has sharp entropic peaks at L = 0, 0.25, 0.5, 0.75, and 1.0, and that these barriers disappear as the number of hidden neurons increases, making learning in larger networks essentially barrier-free. The paper's central quantitative conclusion is about how this entropy curve changes with the number of hidden neurons.

Significance. If the entropy calculations can be made well-defined, the paper offers a useful bridge between statistical-physics density-of-states methods and small neural networks. Strengths include the exhaustive enumeration of the 36 two-dimensional cross-sections, the clean comparison between nonrandomized and randomized batches with an averaging control, and the use of generalized-ensemble sampling rather than raw random sampling. The phase-transition language is mostly framed as an analogy, which is appropriate. However, the paper's central quantitative claims are not yet supported because the sampling domain and Monte Carlo diagnostics are missing; these gaps are fixable and do not require changing the network setup.

major comments (3)

[Sec. III.D, Eq. (8), Fig. 8] The density g(L) is defined through an unspecified phase-space measure DX. The parameter space is unbounded, and with sigmoid activations the loss saturates as any weight or bias diverges, so the Lebesgue integral is not well-defined on R^9 without an explicit cutoff. The text states only that simple sampling and Wang-Landau sampling were used; it does not give the parameter bounds, bin width, number of samples, or Wang-Landau convergence criteria. Consequently the sharp peaks, the 'astonishing' suppression between them, and the disappearance of barriers as nh grows are, with the information provided, properties of an unstated finite box rather than established properties of the network. Please specify the sampling domain and normalization, report the Monte Carlo parameters, and show that the relative peak structure is robust to the box size.
[Sec. III.A, Fig. 2] The claim that the power-law exponent gamma is independent of the learning rate and depends only on nh is not backed by the evidence shown. Fig. 2 contains five curves for nh = 2 and one for nh = 18, all without error bars, multiple seeds, or a description of the fitting procedure. The reference lines labeled 1.0 and 1.2 are not fits with uncertainties. Please provide the fitting method, the number of independent runs, and confidence intervals, or reduce the strength of the claim.
[Sec. III.D, Sec. IV] The connection between S(L) and the difficulty of backpropagation is assumed rather than established. Gradient descent moves along loss gradients and does not sample parameter space with the uniform measure used in Eq. (8), and the trajectories shown in Fig. 6 are restricted two-dimensional gradient-descent paths with the other seven parameters fixed at solution #1. The statement that larger networks are 'essentially barrier-free' therefore goes beyond what the entropy curves can prove. The paper should explicitly label this as a heuristic analogy or supply evidence that the density-of-states volumes correlate with actual dwell times of the optimization dynamics.

minor comments (6)

[Sec. II, Eq. (2)] The notation n(l-1)_i for the number of neurons in the previous layer is confusing because this number does not depend on i, and the text refers to 'neuron j' before j has been defined; please simplify the indexing.
[Sec. III.A] The interval eta in [0.002, 17.484] is given to five significant figures without specifying the convergence criterion (e.g., loss below some threshold within a finite number of epochs) or the run-to-run variability; please report a supported range with error estimates or remove the precise bounds.
[Fig. 2] The reference lines labeled 1.0 and 1.2 are not explained in the caption; please state how the exponents were extracted and what the lines represent.
[Sec. III.C.1] The statement that cross-sections for the other solutions in Table II look different but possess similar main features is unsupported by any figure; please add a representative example or qualify the statement as an observation from preliminary runs.
[Fig. 8] Because the entropy curves are shifted vertically, the ordinate values in Fig. 8 are arbitrary; please report the actual offsets or plot them on a common scale to make the relative normalization meaningful.
[Throughout] Please correct typographical errors such as 'inhowfar', 'entopy', and 'inital parameter settings'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: S(L) is computed directly from a Monte Carlo sampled density of loss, and the peak structure is explained by the loss definition rather than fitted or imported from a self-citation.

full rationale

The paper's central claim is the microcanonical entropy curve S(L)=ln g(L) computed via Eq. (8) and sampled by simple sampling and Wang-Landau/multicanonical Monte Carlo (Sect. II.D, Fig. 8). The loss L is defined in Eqs. (4)-(5) as an average of squared deviations over the four XOR cases, so the appearance of enhanced density near L=0, 0.25, 0.5, 0.75, 1.0 is explicitly linked in the text to the discrete counting of correctly versus incorrectly evaluated cases ('Loss values ~0.25 can typically be attributed to situations, where the XOR logic of one case in the batch is determined wrongly...'), i.e., the paper itself derives these locations from the loss definition rather than presenting them as an unexplained prediction. The disappearance of smaller entropic barriers as nh grows is a comparison of independently computed curves, not a fit of a parameter to a target. The self-citation [14] (Qi and Bachmann) is used only to justify the interpretive analogy that convex entropy regions indicate first-order-like transitions; it is not load-bearing for the computation of g(L) or for the qualitative peak locations. Concerns about the unspecified integration domain and missing convergence diagnostics in Fig. 8 are validation/robustness issues, not circularity: even if the absolute normalization or box dependence changed the curve shapes, the derivation would still be a direct sampling result rather than a consequence of the paper's own assumptions being fed back as the output.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the assumption that the Monte Carlo estimate of g(L) is converged and that 2D cross-sections through one optimal solution are representative. No fit parameters are central, but the power-law exponent and convergence bounds are extracted numbers that support the overclaim.

free parameters (3)

Power-law exponent gamma = ~1.0 (nh=2), ~1.2 (nh=18)
Extracted as the slope of log L vs log tau in Fig. 2; claimed to depend on hidden neuron count but not learning rate, without reported fit error bars.
Learning rate convergence bounds = [0.002, 17.484]
Reported as 'extensive tests with up to five million epochs' for the nh=2 network; an oddly precise interval likely initialization-dependent.
Convergence threshold L < 0.005 = 0.005
Ad hoc threshold used to define epoch of convergence in Fig. 5.

assumptions (4)

domain assumption Density of states / microcanonical entropy definition S(L) = ln g(L) applies to loss landscapes.
Assumes the loss can be treated as an energy and that the uniform measure on parameters is the relevant phase-space measure (Sect. II.D).
domain assumption The 2-hidden-neuron sigmoid network is the simplest capable XOR network and captured the essential learning dynamics of larger networks.
The paper asserts the generalization from this tiny network to larger networks without proof (Sect. I and IV).
domain assumption Monte Carlo sampling of the parameter space converged to the true density of loss.
No convergence tests, parameter bounds, or sampling statistics are reported for Fig. 8 (Sect. II.D).
ad hoc to paper 2D cross-sections through a single optimal solution (solution #1) are representative of the loss landscape.
Seven parameters are fixed at the values of one particular optimum; the paper notes other solutions have similar features but does not quantify this claim (Sect. III.C.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dissecting a Small Artificial Neural Network." pith.science (2026). https://pith.science/paper/BXP3EXLJ

@misc{pith2026250108341,
  author       = {Pith},
  title        = {Pith review of: Dissecting a Small Artificial Neural Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXP3EXLJ}},
  note         = {Machine review of arXiv:2501.08341}
}

read the original abstract

We investigate the loss landscape and backpropagation dynamics of convergence for the simplest possible artificial neural network representing the logical exclusive-OR (XOR) gate. Cross-sections of the loss landscape in the nine-dimensional parameter space are found to exhibit distinct features, which help understand why backpropagation efficiently achieves convergence toward zero loss, whereas values of weights and biases keep drifting. Differences in shapes of cross-sections obtained by nonrandomized and randomized batches are discussed. In reference to statistical physics we introduce the microcanonical entropy as a unique quantity that allows to characterize the phase behavior of the network. Learning in neural networks can thus be thought of as an annealing process that experiences the analogue of phase transitions known from thermodynamic systems. It also reveals how the loss landscape simplifies as more hidden neurons are added to the network, eliminating entropic barriers caused by finite-size effects.

Figures

Figures reproduced from arXiv: 2501.08341 by the authors.

**Figure 2.** FIG. 2. Convergence of the loss function for the minimal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** FIG. 3. Drifts of weights and biases for the (a) hidden and (b) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: FIG. 5. Epoch of convergence [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 4.** Figure 4: FIG. 4. Convergence of activations [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: FIG. 6. Selected cross-sections of the loss landscape throu [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: FIG. 7. Cross-section through the loss landscape with [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: FIG. 8. Microcanonical entropies [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 32 canonical work pages

[14]

M. L. Minsky and S. A. Papert, Perceptrons (MIT Press, Cambridge, MA, USA, 1969)

work page 1969
[1]

0 (0, 0) a(1) 2 a(1) 1 a(2) 1 0

work page
[2]

8 (0, 1) a(1) 2 a(1) 1 a(2) 1 0

work page
[3]

8 (1, 0) a(1) 2 a(1) 1 a(2) 1 0 30000 60000 90000 120000 150000 τ 0

work page
[4]

8 a(l) i (1, 1) a(1) 2 a(1) 1 a(2) 1 FIG. 4. Convergence of activations a(l) i of hidden and output neurons to solution #1 in Table II for all four cases listed in Table I as functions of epoch τ for the XOR sigmoid network. The learning rate was η = 0. 1. if ∆ w(1) 11 < 2. 2, ∆ w(1) 12 < 2. 2, or ∆ w(2) 11 < 1. 6 (provided in each case all other paramete...

work page
[5]

highest-entropy

Nonrandomized Batch The loss was deﬁned in Eq. (4) as an average of the square deviations of the actual from the expected output values over the batch of all input cases for any given set of parameter values. Therefore, the loss function depends on all parameters and can be interpreted as a landscape in this embedding space. For the network we chose, the ...

work page
[6]

ﬁnite- size

Randomized Batches Randomized batches are typically used for the learn- ing process of large neural networks, where working with the full batch is not economical. It is a necessity if mini- batches are used. In this section, we again use the full batch, but with randomized cases, to determine cross- sections through the loss landscape. Since the results s...

work page
[7]

0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 L 0 10 20 30 40 50S(L) nh = 18 14 10 6 2 1 FIG. 8. Microcanonical entropies S(L) for XOR networks with diﬀerent numbers of neurons in the hidden layer, nh. The curves have been shifted relative to each other vertical ly for better visibility. from the smoothing of the entopy curves for the larger networks were observed in t...

work page

Show all 35 references

[8]

H. J. Kelley, ARS Journal 30, 947 (1960)

1960
[9]

A. E. Bryson, A gradient method for optimizing multi- stage allocation processes , in: Proc. Harvard Symposium on Digital Computers and Their Applications (1962)

1962
[10]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Na- ture 323, 533 (1986)

1986
[11]

Werbos, Proc

P. Werbos, Proc. IEEE 78, 1550 (1990)

1990
[12]

Russell and P

S. Russell and P. Norvig, Artiﬁcial Intelligence: A Mod- ern Approach, Fourth Edition (Pearson, 2022)

2022
[13]

Rosenblatt, Psychol

F. Rosenblatt, Psychol. Rev. 65, 386 (1958)

1958
[15]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning Internal Representations by Error Propagation , in: Parallel Distributed Processing, Volume 1: Explo- rations in the Microstructure of Cognition: Foundations , ed. by D. E. Rumelhart, J. L. McClelland, and the PDP Res...

1986
[16]

Herberg, e-print: arXiv:2304.05133 (2023)

E. Herberg, e-print: arXiv:2304.05133 (2023)

2023 arXiv
[17]

J. J. Hopﬁeld, Proc. Nat. Acad. Sci. (USA) 79, 2554 (1982)

1982
[18]

Poggioa, A

T. Poggioa, A. Banburskia, and Q. Liaoa, Proc. Nat. Acad. Sci. (USA) 117, 30039 (2020)

2020
[19]

M. Gori, R. Franzosi, G. Pettini, and M. Pettini, J. Phys. A: Math. Theor. 55, 375002 (2022). 12

2022
[20]

Di Cairano, J

L. Di Cairano, J. Phys. A: Math. Theor. 55, 27LT01 (2022)

2022
[21]

Qi and M

K. Qi and M. Bachmann, Phys. Rev. Lett. 120, 180601 (2018)

2018
[22]

Aierken and M

D. Aierken and M. Bachmann, PCCP 25, 30246 (2023)

2023
[23]

Bachmann, Thermodynamics and Statistical Mechan- ics of Macromolecular Systems (Cambridge University Press, Cambridge UK, 2014)

M. Bachmann, Thermodynamics and Statistical Mechan- ics of Macromolecular Systems (Cambridge University Press, Cambridge UK, 2014)

2014
[24]

A. J. Ballard, R. Das, S. Martiniani, D. Mehta, L. Sagun, J. D. Stevenson, and D. J. Wales, Phys. Chem. Chem. Phys. 19, 12585 (2017)

2017
[25]

Mehta, X

D. Mehta, X. Zhao, E. A. Bernal, and D. J. Wales, Phys. Rev. E 97, 052307 (2018)

2018
[26]

H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, arXiv:1712.09913 (2018)

2018 arXiv
[27]

Mathematical Aspects of Deep Learning , ed. by P. Grohs and G. Kutyniok (Cambridge University Press, Cam- bridge UK, 2022)

2022
[28]

Whitelam, V

S. Whitelam, V. Selin, I. Benlolo, C. Casert, and I. Tam- blyn, arXiv:2205.07408 (2022)

2022 arXiv
[29]

D. P. Kingma and J. L. Ba, Adam: A Method for Stochas- tic Optimization, in: Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA, May 7-9, 2015, ed. by Y. Bengio and Y. LeCun; e-print: arXiv:1412.6980 (2014)

2014 arXiv
[30]

Wang and D

F. Wang and D. P. Landau, Phys. Rev. Lett. 86, 2050 (2001)

2001
[31]

Wang and D

F. Wang and D. P. Landau, Phys. Rev. E 64, 056101 (2001)

2001
[32]

B. A. Berg and T. Neuhaus, Phys. Lett. B 267, 249 (1991)

1991
[33]

B. A. Berg and T. Neuhaus, Phys. Rev. Lett. 68, 9 (1992)

1992
[34]

Draxler, K

F. Draxler, K. Veschgini, M. Salmhofer, and F. Ham- precht, Essentially No Barriers in Neural Network En- ergy Landscape , in: Proceedings of the 35th Interna- tional Conference on Machine Learning, PMLR 80, 1309 (2018)

2018
[35]

D. H. E. Gross, Microcanonical Thermodynamics: Phase Transitions in “Small” Systems (World Scientiﬁc, Singa- pore, 2001)

2001

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [14]

M. L. Minsky and S. A. Papert, Perceptrons (MIT Press, Cambridge, MA, USA, 1969)

work page 1969

[2] [1]

0 (0, 0) a(1) 2 a(1) 1 a(2) 1 0

work page

[3] [2]

8 (0, 1) a(1) 2 a(1) 1 a(2) 1 0

work page

[4] [3]

8 (1, 0) a(1) 2 a(1) 1 a(2) 1 0 30000 60000 90000 120000 150000 τ 0

work page

[5] [4]

8 a(l) i (1, 1) a(1) 2 a(1) 1 a(2) 1 FIG. 4. Convergence of activations a(l) i of hidden and output neurons to solution #1 in Table II for all four cases listed in Table I as functions of epoch τ for the XOR sigmoid network. The learning rate was η = 0. 1. if ∆ w(1) 11 < 2. 2, ∆ w(1) 12 < 2. 2, or ∆ w(2) 11 < 1. 6 (provided in each case all other paramete...

work page

[6] [5]

highest-entropy

Nonrandomized Batch The loss was deﬁned in Eq. (4) as an average of the square deviations of the actual from the expected output values over the batch of all input cases for any given set of parameter values. Therefore, the loss function depends on all parameters and can be interpreted as a landscape in this embedding space. For the network we chose, the ...

work page

[7] [6]

ﬁnite- size

Randomized Batches Randomized batches are typically used for the learn- ing process of large neural networks, where working with the full batch is not economical. It is a necessity if mini- batches are used. In this section, we again use the full batch, but with randomized cases, to determine cross- sections through the loss landscape. Since the results s...

work page

[8] [7]

0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 L 0 10 20 30 40 50S(L) nh = 18 14 10 6 2 1 FIG. 8. Microcanonical entropies S(L) for XOR networks with diﬀerent numbers of neurons in the hidden layer, nh. The curves have been shifted relative to each other vertical ly for better visibility. from the smoothing of the entopy curves for the larger networks were observed in t...

work page