Pith. sign in

REVIEW 5 major objections 5 minor 17 references

On the Mathematical Impossibility of Safe Universal Approximators

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Any universal approximator capable of useful real-world performance must contain dense, unavoidable catastrophic failures, so perfect alignment is mathematically impossible.

desk verdict A provocative reframing of AI safety as mathematically impossible, but the central theorem is a non-sequitur and the quantitative sandwich is hand-waved; the paper is a useful discussion piece, not a proof. read the letter →

arxiv 2507.03031 v1 pith:B4HLDPC5 submitted 2025-07-03 cs.LG cs.AI

Jasper Yao This is my paper
classification cs.LGcs.AI MSC 68T0758K35
keywords universalapproximationtheoremAIalignmentcatastrophicfailuressingularitytheoryadversarialexamplesFisherinformationmatrixneuralnetworksafetyimpossibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Universal approximation is the property that lets a computational system approximate any continuous function, and it is the mathematical basis of modern neural networks. This paper argues that the same property makes perfect safety impossible: any universal approximator powerful enough to be useful must contain a dense set of inputs at which tiny perturbations cause large behavioral changes. The argument has three layers: a combinatorial proof for piecewise-linear networks, a singularity-theoretic proof for all universal approximators, and an information-geometric claim that real-world tasks force networks into unstable Fisher information structures. If the proof is correct, alignment is not an engineering problem to be solved but a mathematical constraint to be managed, and the only remaining choices are limiting capability, accepting statistical rather than guaranteed safety, or abandoning traditional epsilon-delta safety standards.

What carries the argument

The carrying object is the catastrophe density $\rho(\delta,C)$: the measure of inputs at which a perturbation of size $\delta$ changes the output by more than a tolerance $\varepsilon$. The paper proves an exponential decay bound for the safe region, $\mu(\{x: f \text{ is }(\varepsilon,\delta)\text{-safe at }x\}) \le K\exp(-\alpha C/\delta^d)$, where $C$ is complexity and $d$ the input dimension, and combines it with an information-theoretic lower bound on $C_{\min}$ so that the "sandwich" $C_{\min} \gg C_0$ follows. A second mechanism is a measure transfer: a probability measure on network parameters is assumed to induce, in the large-complexity limit, the generic measure on the space of all smooth functions, so that the singularity-theoretic statement "almost all smooth functions have dense catastrophes" becomes "almost all sufficiently complex networks have dense catastrophes." A third mechanism is the Fisher Information Matrix, whose pathologically large eigenvalue ratios are claimed to make the natural gradient explosive and thereby guarantee instability.

What would settle it

Take a trained network with complexity far above the paper's $C_0$, fix the paper's $\varepsilon$ and $\delta$, and estimate the proportion of inputs in a dense grid that are $(\varepsilon,\delta)$-safe. If even half the points are safe, the claimed exponential bound $\mu(\text{safe}) \le K\exp(-\alpha C/\delta^d)$ is violated for that network and the central quantitative claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that dense catastrophic behavior is a necessary cost of useful universal approximation. It is stated as the Universal Approximator Catastrophe Theorem: for any universal approximator $U$ that achieves useful performance on real-world tasks, catastrophic behavioral failures are mathematically inevitable. The proof binds together three necessities: in piecewise-linear networks the number of linear regions (and hence the boundaries where gradients change) grows exponentially with complexity, making expressive power and catastrophe density the same quantity; in any universal approximator, the ability to approximate all continuous functions includes the generic functions that singularity theory says are dense with singularities; and real-world tasks force a pathological Fisher information spectrum (eigenvalue ratios of order $10^6$ to $10^8$) that guarantees explosive sensitivity in some directions. The quantitative core is the "Impossibility Sandwich": the minimum complexity $C_{\min}$ needed for usefulness exceeds the maximum complexity $C_0$ allowed for safety, with the paper's estimates placing $C_0$ near $10^{-5}$ for standard choices of $\varepsilon,\delta$ in two dimensions while practical networks have $10^8$ to $10^{12}$ parameters.

Load-bearing premise

The load-bearing premise is that a random network drawn from a large parameter space behaves, in the limit, like a random draw from the space of all smooth functions, so that what is true of almost all smooth functions transfers to almost all sufficiently complex networks.

Editorial extensions

If this is right

  • Perfect alignment of a useful universal approximator is impossible; the only available goals are statistical control, capability limits, or abandonment of $\varepsilon,\delta$ safety guarantees.
  • The safe-complexity ceiling $C_0$ is so low that no nontrivial network can satisfy it, while practical systems exceed it by factors of $10^{13}$ to $10^{17}$.
  • Any system able to verify, detect, interpret, or improve the safety of a universal approximator would itself be a universal approximator and would inherit the same dense failures.
  • The three-level argument covers every major architecture in use, including feedforward, convolutional, recurrent, transformer, and graph networks, so no architectural variation escapes the conclusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same argument, if correct, extends to any learned system that contains a universal approximating component, including generative models, agentic systems, and hybrid symbolic-neural systems, even if the paper does not list them.
  • A direct experimental test of the paper's quantitative claim would measure empirical catastrophe density on a large trained network and compare it to the exponential bound; the paper itself treats adversarial examples as evidence but does not run this measurement.
  • A further implicit consequence is that safety guarantees should be reformulated as distribution-dependent bounds on the probability of triggering latent failures, rather than guarantees over all inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper claims to prove a sweeping impossibility result: any universal approximator complex enough to be useful on real-world tasks must exhibit a dense set of catastrophic failure points, making perfect alignment or provable safety mathematically impossible. The argument is organized into three "pillars": a combinatorial analysis of piecewise-linear (ReLU) networks, a topological argument based on Whitney's singularity theorem applied to the function space of all representable functions, and an "information geometric" argument linking task complexity to pathological Fisher Information Matrix spectra. These are combined into an "Impossibility Sandwich" (C_min >> C_0) and a set of corollaries that declare impossibility of catastrophe detection, safety verification, interpretability, recursive self-improvement, and AI safety research itself. The manuscript concludes that alignment should be reframed as operating under irreducible uncontrollability rather than eliminating it.

Significance. If the central theorem were proven rigorously, the result would be of major significance for the foundations of AI safety, as it would place hard mathematical limits on reliable control of neural systems. The paper also engages with a genuine scientific question: whether universal approximation guarantees coexist with stability or safety properties. However, the manuscript as written falls far short of establishing its claims. The proofs are mostly sketches that rely on unstated and nontrivial measure-theoretic and topological assumptions, and several steps are non-sequiturs. I see no salvageable central theorem in the current text; the correct recommendation is rejection.

major comments (5)
  1. [§3.1, Theorem 3.1] The proof sketch asserts that a probability measure on network parameters induces a measure on F_C that, as C grows, converges to a measure on F_∞ preserving the full-measure set of Whitney-generic functions. This convergence is not stated as a lemma, no topology on F_∞ is specified, and no argument is given that pushforwards of parameter distributions concentrate on generic functions rather than on the degenerate image of the parameterization. Without this, the conclusion that "almost all sufficiently complex networks" have dense catastrophes is unsupported. The proof sketch explicitly invokes the convergence as if it were automatic, which it is not.
  2. [§3.2, steps 2–3] The step from "adversarial examples exist" to "real-world task functions are catastrophe-ridden" is a non-sequitur. Adversarial examples characterize a learned model's sensitivity in off-manifold directions and occur even for simple linear classifiers (Goodfellow et al., 2014), so they do not demonstrate that the ground-truth labeling function has any singularities. Moreover, universal approximation does not force the approximator to reproduce singularities of the target: for instance, a smooth tanh network can approximate |x| on a compact set arbitrarily well while remaining C^1, so target-function singularities need not be implemented as input-space catastrophes of the approximator. The theorem never proves a link between target singularities and (ε,δ)-catastrophes at the same locations.
  3. [§5.1–5.5, Theorems 5.1 and 5.3] The quantitative bounds are asserted without derivation of the constants. In Theorem 5.1, the geometric constants c_k are never defined or computed, so the claimed "exact" bound is not a theorem but a template. The formula in Theorem 5.3 changes exponent from d to d−1 without justification, and the proof sketch does not derive the explicit α = ln(2)/Γ(d+1). Example 5.2 reports a numerical value (≈0.43) without showing how the constants are obtained, and Example 5.4 contains a broken sentence ("99") and proceeds to compute C_0 ≈ 10^-5 for a 2D input, which is not credible for a safety threshold. The "Impossibility Sandwich" (§5.4) consequently has no quantitative foundation.
  4. [§3.4, Theorem 3.5] The proof is a chain of assertions: (1) FIM pathology is an empirical fact, (2) task complexity forces pathology, (3) pathology necessitates behavioral catastrophes. None of these implications is proven. No theorem shows that high mutual information I(X;Y) forces an extreme eigenvalue spectrum, and no theorem connects a pathological FIM to dense (ε,δ)-catastrophes in input space. The mention of natural gradient explosiveness is irrelevant to input-space behavioral instability unless the parameter-to-function map is shown to transfer the instability, which is not done. The claim of a "complete causal chain" is therefore unjustified.
  5. [§5.9, Corollary 3.4, Theorem 3.2] The paper's own reconciliation in §5.9 undercuts the density claim. It states that catastrophes are "often latent" because the data manifolds on which networks are tested are lower-dimensional, so a random input is unlikely to trigger a catastrophe. But Corollary 3.4 and Theorem 3.2 claim that the safe-region measure is exponentially small and that almost every point is a potential failure point. If the high-dimensional input space is the relevant domain, the density claim conflicts with the acknowledged practical usefulness on low-dimensional data manifolds; if the data manifold is the relevant domain, the theorems' statements are misleading. The paper never formalizes the relationship between the ambient input space and the data manifold, nor does it reconcile the two claims.
minor comments (5)
  1. [§3, preamble] The central quantity "catastrophe density ρ(δ)" is used before it is formally defined; the definition should be stated precisely in Section 3 and then used consistently.
  2. [§5.4, Example 5.4] The sentence "Let's assume a 2D input space (d=2) and a desired safety level of ρ_max = 0.01 (99" is truncated and unclear; it should be completed.
  3. [§4.2.2] The assertion that CNNs with pooling achieve universal approximation for translation-invariant functions is stated without a citation; a reference would be needed for this architectural coverage claim.
  4. [§8.1] In the ε-δ Alignment Trilemma, the first option "Accept Dense Failures" uses δ both as a safety parameter and as the threshold in ρ_cat > 1−δ, which is confusing; the notation should be disentangled.
  5. [General] The paper's abstract and introduction repeat the same claims verbatim, and the conclusion restates the introduction nearly word-for-word; tightening the exposition would improve clarity even in a revision.

Circularity Check

5 steps flagged · score 8.0 of 10

Central claim reduces to its own inputs: Pillar 1 defines 'catastrophes' as ReLU region boundaries (so expressive power = catastrophe density by construction); Theorem 3.5 restates the cited FIM observation as a necessity theorem; Theorem 3.2 treats adversarial examples in trained models as 'proof' that useful models must be catastrophic; Theorem 3.1 assumes its own conclusion.

  1. self definitional [Section 5.1 (Theorem 5.1), Section 9 conclusion, Abstract Pillar 1]
    "For a ReLU network, the catastrophic boundaries are the hyperplanes defined by each neuron. The density of catastrophes can be bounded by analyzing the volume of the input space near these hyperplanes and their intersections. — And, in the conclusion: The piecewise-linear nature of modern networks means their expressive power is mathematically equivalent to the density of their catastrophic boundaries."

    ReLU-network expressive power is here counted by the number of linear regions, and 'catastrophic boundaries' are defined as the boundaries of those regions (the neurons' hyperplanes). Both sides of the claimed proportionality — 'expressive power is mathematically equivalent to the density of their catastrophic boundaries' — are therefore the same geometric object, so Pillar 1's 'Combinatorial Necessity' holds by construction. The unargued interpretive leap is labeling a gradient-direction change at a continuous ReLU boundary a 'catastrophic behavioral failure'; once that label is adopted, Theorem 5.1 and Theorem 3.3's bounds (and the C_0 threshold built on them) merely restate the boundary-count definition.

  2. fitted input called prediction [Section 3.2, Theorem 3.2, proof steps 2 and 4]
    "Real-World Tasks are Catastrophic: The universal existence of adversarial examples is empirical proof that the functions required to solve real-world tasks are not simple or smooth, but are themselves complex and catastrophe-ridden. To be useful, a model must learn these complex, sensitive decision boundaries. ... The empirical evidence shows that this capability is not just theoretical, but strictly necessary for performance. Therefore, any useful universal approximator must exhibit dense catastrophic failures."

    Adversarial examples are observed properties of trained models, not of ground-truth task functions. The proof nonetheless converts their 'universal existence' into 'empirical proof' that the tasks are catastrophe-ridden, and then concludes that a useful model 'must exhibit dense catastrophic failures.' The output of the theorem (models necessarily harbor dense catastrophes) is the input observation (models exhibit adversarial sensitivity) with 'necessarily' attached; the bridge premise — usefulness requires learning such decision boundaries — is asserted, never derived.

3 more flagged steps
  1. fitted input called prediction [Section 3.4, Theorem 3.5 and its proof]
    "1. Empirical Fact: FIM is Pathological in Practice. It is an established empirical result that all high-performing networks trained on real-world tasks (in vision, language, etc.) exhibit a pathological FIM eigenvalue spectrum [4]. 2. Causal Mechanism: Task Complexity Forces Pathology. This is not an accident. The complexity of real-world tasks (high mutual information I(X; Y )) requires the network to learn highly specialized and sensitive parameter configurations."

    The theorem's conclusion — 'Neural networks trained on real-world tasks necessarily develop Fisher Information Matrices with pathological eigenvalue spectra' — reproduces its first premise, the 'established empirical result' cited from [4], with 'necessarily' inserted. The only derivation offered is premise 2, which asserts a causal mechanism (task complexity forces spectral pathology) without any calculation: no bound links I(X;Y) to the FIM eigenvalue distribution. The input (observed pathology) is thus renamed as the output (necessary pathology).

  2. other [Section 3.1, Theorem 3.1 proof sketch]
    "We can define a measure on the space of network parameters, which induces a measure on the function space FC. As C grows, this measure converges to a measure on F∞. Since the set of functions with dense singularities is a full-measure set in F∞, the probability of selecting a network that corresponds to a function in this set approaches 1."

    The theorem's conclusion — that almost every sufficiently complex network has catastrophe density approaching 1 — is delivered entirely by the premise that the parameter-induced measure on F_C converges, as C grows, to a measure on F_∞ that carries the full-measure set of Whitney-generic singularity-dense functions. That convergence is exactly what the theorem needs to prove; it is not established for any concrete parameter distribution, and approximation in function space does not imply that random parameter draws track generic singularities rather than smooth approximations of them.

  3. other [Section 5.4, Impossibility Sandwich (Example 5.4 and 5.5)]
    "A complex task (e.g., modeling English) has a high mutual information I(X; Y ) between inputs and outputs. To learn this task, the network's parameter space must be large enough to represent this information. This provides a hard, architecture-independent lower bound on Cmin. ... The conclusion is inescapable: Cmin ≫ C0"

    The almost-inescapable sandwich ordering C_min ≫ C_0 is assembled from an asserted lower bound and a definitional upper bound. C_0 is obtained by plugging freely chosen safety constants (ρ_max = 0.01, δ = 10^-3, giving C_0 ≈ 2.8 × 10^-5) into the boundary-density formula of Theorem 5.3, which itself restates the definitional identification of Section 5.1; with δ = 10^-3 the threshold collapses below a single parameter, so any real network violates it by construction. C_min, by contrast, is never derived: no inequality linking task mutual information I(X;Y) to a parameter count is stated, and the claimed 'hard, architecture-independent lower bound' is an assertion. The paper's strongest quantitative conclusion is therefore its thesis re-stated rather than a consequence of the stated bounds.

full rationale

The paper's headline claim does not follow from an independent derivation; its pillars each reduce to their own inputs. Pillar 1 (Combinatorial Necessity) is definitional: Section 5.1 declares 'the catastrophic boundaries are the hyperplanes defined by each neuron,' and expressive power for piecewise-linear nets is counted by the same linear regions, so 'expressive power is mathematically equivalent to the density of their catastrophic boundaries' (Section 9) is an identity, not a result. Pillar 3 (Empirical Necessity) is observation-renaming: Theorem 3.5's conclusion is the 'established empirical result' it cites from [4], and Theorem 3.2 step 2 takes adversarial examples — a property of trained models — as 'empirical proof' that ground-truth tasks are catastrophic, then concludes models must be catastrophic; the argument's input is its output. Theorem 3.1, which carries the 'almost all networks' claim, does not prove but assumes that parameter-induced measures converge to the Whitney-generic measure on F_∞; the proof sketch states this convergence as its premise, and it is only a proof sketch. There is no author self-citation chain here (no prior Yao work is cited), so the self-citation patterns do not apply; the problem is instead that the load-bearing external citations ([2], [4]) are used as premises identical to the conclusions they are said to establish. The paper's own Section 5.9 concedes that 'the catastrophic failures predicted by the theory do not prevent usefulness because they are often latent,' since 'data manifolds on which networks are typically tested are much lower-dimensional' — admitting that the dense catastrophe set and the useful-operation set can be disjoint, which undercuts the necessity of dense failures for usefulness. Missing-support flags: Theorem 3.1, Theorem 3.3, and Theorem 5.3 are only 'proof sketches,' Theorem 4.1 is declared 'immediate by the Universal Approximator Catastrophe Theorem,' and the invocations of Whitney's and Mather's theorems are misapplications of external results, which is a correctness risk rather than circularity. Because the central quantitative claims — Combinatorial Necessity, the FIM theorem, and the C_min ≫ C_0 sandwich built on them — are forced by definition or by renaming observed model pathologies as necessities, the score is 8.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on standard approximation and singularity theorems, plus several assumptions specific to this paper: measure convergence on function spaces, a causal reading of adversarial examples, and an asserted link between FIM eigenvalue ratios and behavioral instability. The quantitative results also rely on constants c_k and alpha that are never given values.

free parameters (4)
  • c_k geometric constants
    Introduced in Theorem 5.1 for the ReLU catastrophe density bound; no values or derivations are given, so the bound is not computable.
  • alpha (architecture-dependent constant) = alpha approx log(m) in Theorem 3.3; alpha = ln(2)/Gamma(d+1) in Theorem 5.3
    Two conflicting expressions are given with no derivation connecting them; the numerical examples use the second.
  • K (normalization constant) = 1
    Set to 1 by assumption in Theorem 3.3 to simplify the bound; not derived from the geometry.
  • Cmin (minimum useful complexity)
    Assumed to be large because I(X;Y) is large; no quantitative relationship is derived, so the sandwich inequality is asserted.
assumptions (5)
  • standard math Whitney's theorem on genericity of singularities
    Invoked in Theorem 3.1 and 3.2 to claim almost every smooth function is dense with catastrophic singularities; the theorem concerns generic properties of mappings, and the paper does not establish that this translates to dense catastrophic points for network functions.
  • standard math Universal Approximation Theorem
    Used to say F_C becomes dense in F_infty and that any useful network can represent catastrophic functions; the theorem states approximation in function space, not measure convergence over parameter space.
  • ad hoc to paper Measure convergence of parameter-induced distributions to a full-measure function-space distribution
    The Theorem 3.1 proof sketch asserts this convergence without proof; it is the load-bearing step for "almost all sufficiently complex networks have dense catastrophes."
  • ad hoc to paper Adversarial examples demonstrate that real-world tasks are themselves catastrophic
    Used in Pillar 3 and Theorem 3.2; this equates a model vulnerability with task-intrinsic complexity without independent evidence.
  • ad hoc to paper Pathological FIM spectra mathematically force behavioral catastrophes
    Theorem 3.5 states this as a mathematical consequence, but only gives an intuitive argument about natural gradient norms; no theorem or bound is shown.
invented entities (1)
  • catastrophe density rho(delta) and (epsilon, delta)-safe regions
    purpose: Formalizes the measure of input space where small perturbations cause large behavior changes; the paper's central quantity.
    No rigorous definition is given, and the quantitative bounds depend on unspecified constants; the entity is defined only through loose prose and equations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Mathematical Impossibility of Safe Universal Approximators." pith.science (2026). https://pith.science/paper/B4HLDPC5

@misc{pith2026250703031,
  author       = {Pith},
  title        = {Pith review of: On the Mathematical Impossibility of Safe Universal Approximators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4HLDPC5}},
  note         = {Machine review of arXiv:2507.03031}
}
read the original abstract

We establish fundamental mathematical limits on universal approximation theorem (UAT) system alignment by proving that catastrophic failures are an inescapable feature of any useful computational system. Our central thesis is that for any universal approximator, the expressive power required for useful computation is inextricably linked to a dense set of instabilities that make perfect, reliable control a mathematical impossibility. We prove this through a three-level argument that leaves no escape routes for any class of universal approximator architecture. i) Combinatorial Necessity: For the vast majority of practical universal approximators (e.g., those using ReLU activations), we prove that the density of catastrophic failure points is directly proportional to the network's expressive power. ii) Topological Necessity: For any theoretical universal approximator, we use singularity theory to prove that the ability to approximate generic functions requires the ability to implement the dense, catastrophic singularities that characterize them. iii) Empirical Necessity: We prove that the universal existence of adversarial examples is empirical evidence that real-world tasks are themselves catastrophic, forcing any successful model to learn and replicate these instabilities. These results, combined with a quantitative "Impossibility Sandwich" showing that the minimum complexity for usefulness exceeds the maximum complexity for safety, demonstrate that perfect alignment is not an engineering challenge but a mathematical impossibility. This foundational result reframes UAT safety from a problem of "how to achieve perfect control" to one of "how to operate safely in the presence of irreducible uncontrollability," with profound implications for the future of UAT development and governance.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

  1. [2]

    Explaining and harnessing adver- sarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6572, 2014

    arXiv 2014

  2. [3]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. volume 2, pages 359–366. Elsevier, 1989

    work page 1989

  3. [4]

    Universal statistics of fisher information in deep neural networks: Mean field approach

    Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In Artificial Intelligence and Statistics, pages 1004–1012. PMLR, 2019

    work page 2019

  4. [5]

    Catastrophic forgetting in continual graph learning

    Huihui Liu, Yiding Yang, and Xinchao Wang. Catastrophic forgetting in continual graph learning. arXiv preprint, 2020

    work page 2020

  5. [6]

    Stability of c ∞ mappings, ii

    John N Mather. Stability of c ∞ mappings, ii. infinitesimal stability implies stability. Annals of Mathematics, 89(2):254–291, 1969

    work page 1969

  6. [8]

    On the Nonlinearity of Layer Normalization

    Yunhao Ni, Yuxin Lei, Zirui Cai, Tong Zhang, Ruitu Wang, and Junfeng Chen. On the nonlinearity of layer normalization. arXiv preprint arXiv:2406.01255, 2024. URL https: //arxiv.org/abs/2406.01255

    work page Pith review arXiv 2024

  7. [9]

    Graph neural networks exponentially lose expressive power for node classification

    Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations, 2020

    work page 2020

  8. [10]

    Classes of recursively enumerable sets and their decision problems

    Henry Gordon Rice. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2):358–366, 1953

    work page 1953

Show all 17 references
  1. [11]

    Bridgeout: stochastic bridge regularization for deep neural networks

    Amir Salehi et al. Bridgeout: stochastic bridge regularization for deep neural networks. In International Conference on Machine Learning , 2018. URL https://arxiv.org/ abs/1804.08042. 16

    2018 arXiv

  2. [12]

    Stabilize deep resnet with a sharp scaling factor τ

    Xiang Sun, Hengyue Hu, Qingshan Zhang, and Degang Peng. Stabilize deep resnet with a sharp scaling factor τ. Machine Learning, 111:3633–3659, 2022. URL https://link. springer.com/article/10.1007/s10994-022-06192-x

    2022 doi

  3. [13]

    On the training instability of shuffling sgd with batch normalization

    David X Wu, Nika Makarova, Preetum Nakkiran, Sadhika Malladi, Sanjeev Arora, and Moritz Hardt. On the training instability of shuffling sgd with batch normalization. In In- ternational Conference on Machine Learning, 2023. URL https://arxiv.org/abs/ 2302.12444

    2023 arXiv

  4. [14]

    On the equivalence between graph isomorphism testing and func- tion approximation with gnns

    Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. On the equivalence between graph isomorphism testing and func- tion approximation with gnns. In Advances in Neural Information Processing Systems , volume 32, 2019. URL https:...

    2019

  5. [15]

    Are transformers universal approximators of sequence-to-sequence functions? In Inter- national Conference on Learning Representations (ICLR), 2020

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Ku- mar. Are transformers universal approximators of sequence-to-sequence functions? In Inter- national Conference on Learning Representations (ICLR), 2020. URL https://arxiv. org/abs/1912.10077

    2020 arXiv

  6. [16]

    Stabilizing transformer training by preventing attention entropy collapse

    Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the 40th International Conference on Machine Learning , pages 40770– 40803...

    2023 arXiv

  7. [17]

    Dropout in training neural networks: Flatness of solution and noise structure

    Zhanran Zhang et al. Dropout in training neural networks: Flatness of solution and noise structure. arXiv preprint arXiv:2111.01022 , 2021. URL https://arxiv.org/abs/ 2111.01022. 17

    2021 arXiv

  8. [2023]

    URL https://arxiv.org/abs/2306.12929

    arXiv

  9. [2025]

    URL https://arxiv.org/abs/2501.19399

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.