RECORD
Deep Learning is Singular, and That's Good
Machine Learning · Submitted 22 Oct 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. | background 1.00 | 0.06 | 0.10 | asserted 1.00 |
| s1 | This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. | background 0.81 | 0.06 | 0.14 | asserted 1.00 |
| s2 | Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. | gap 0.95 | 0.07 | 0.09 | asserted 0.95 |
| s3 | Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice. | contribution 0.85 | 0.74 | 0.51 | hedged possibility 0.53 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.
Human review
How the literature uses it
4 arXiv papers cite this in 6 sentences
Papers citing it · 5 sentences without a confident label
Cites it as background 1 paper
-
For an introduction to singular learning theory in the context of deep learning see [ref] .
Geometry of Program Synthesis · Preliminaries · 0.97
What it cites · 10 sentences without a confident label
Contests 1 paper
-
Our experiments further confirm that the Laplace approximation of the predictive distribution [ref] is not only theoretically inappropriate but performs poorly.
A Bayesian Perspective on Generalization and Stochastic Gradient Descent · Introduction · contest 0.77
Cites as support 1 paper
-
The competitive performance of SiLU on standard benchmarks [ref] shows that the non-analyticity of ReLU is probably not fundamental.
Searching for Activation Functions · Simple functions and complex singularities · 0.98
Cites as background 2 papers
-
It is folklore in the deep learning community that flatness of minima is related to generalisation [ref] and this claim has been revisited in recent years [ref] .
A Bayesian Perspective on Generalization and Stochastic Gradient Descent · Volume dimension, effective degrees of freedom, and flatness · 0.96
-
It is folklore in the deep learning community that flatness of minima is related to generalisation [ref] and this claim has been revisited in recent years [ref] .
Three Factors Influencing Minima in SGD · Volume dimension, effective degrees of freedom, and flatness · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.