RECORD
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Machine Learning · Submitted 20 Jun 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate. This distinguishes linear models with batch normalization from those without batch normalization in terms of both the type of implicit bias and the convergence rate. We further extend our result to a class of two-layer, single-filter linear convolutional neural networks, and show that batch normalization has an implicit bias towards a patch-wise uniform margin. Based on two examples, we demonstrate that patch-wise uniform margin classifiers can outperform the maximum margin classifiers in certain learning problems. Our results contribute to a better theoretical understanding of batch normalization.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | We study the implicit bias of batch normalization trained by gradient descent. | contribution 0.94 | 0.82 | 0.04 | asserted 0.52 |
| s1 | We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate. | result 1.00 | 0.95 | 0.63 | asserted 1.00 |
| s2 | This distinguishes linear models with batch normalization from those without batch normalization in terms of both the type of implicit bias and the convergence rate. | interpretation 0.95 | 0.85 | 0.21 | asserted 1.00 |
| s3 | We further extend our result to a class of two-layer, single-filter linear convolutional neural networks, and show that batch normalization has an implicit bias towards a patch-wise uniform margin. | result 0.75 | 0.95 | 0.21 | asserted 1.00 |
| s4 | Based on two examples, we demonstrate that patch-wise uniform margin classifiers can outperform the maximum margin classifiers in certain learning problems. | result 0.82 | 0.88 | 0.10 | asserted 0.97 |
| s5 | Our results contribute to a better theoretical understanding of batch normalization. | contribution 0.60 | 0.46 | 0.06 | asserted 0.95 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin classifier on the training data with an $\exp(-\Omega(\log^2 t))$ convergence rate. This distinguishes linear models with batch normalization from those without batch normalization in terms of both the type of implicit bias and the convergence rate. We further extend our result to a class of two-layer, single-filter linear convolutional neural networks, and show that batch normalization has an implicit bias towards a patch-wise uniform margin. Based on two examples, we demonstrate that patch-wise uniform margin classifiers can outperform the maximum margin classifiers in certain learning problems. Our results contribute to a better theoretical understanding of batch normalization.
Human review
How the literature uses it
1 arXiv paper cite this in 2 sentences
What it cites · 6 sentences without a confident label
Compares with 1 paper
-
Although [ref] showed that gradient descent converges to a KKT point of the maximum margin problem, the KKT points are usually not unique.
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks · Introduction · 0.80
Cites as background 5 papers
-
When the BN is absent, the implicit bias of gradient descent for the linear model (or deep linear models) is widely studied [ref] : when the data is linearly separable, gradient descent on the cross-entropy loss will converge to the maximum margin solution.
Gradient descent aligns the layers of deep linear networks · Introduction · 0.93
-
From a theoretical perspective, a series of works have established the close connection between the normalization methods and the adaptive effective learning rate [ref] .
Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets · Additional Related Work · 0.92
-
Many normalization methods, including batch normalization, weight normalization [ref] , layer normalization [ref] , and group normalization [ref] , have been recently developed to improve generalization performance.
Layer Normalization · Additional Related Work · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.