Rethinking Normalization and Elimination Singularity in Neural Networks

Alan Yuille; Chenxi Liu; Huiyu Wang; Siyuan Qiao; Wei Shen

RECORD

Rethinking Normalization and Elimination Singularity in Neural Networks

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille

Computer Vision and Pattern Recognition · Submitted 21 Nov 2019

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim To address this issue, we propose BatchChannel Normalization (BCN), which uses batch knowledge to avoid the elimination singularities in the training of channel-normalized models.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID1911.09738 (version 1)

Pith IDMINPENK3, Pith's permanent record of this paper and its review

CitationAlan Yuille, Chenxi Liu, Huiyu Wang, Siyuan Qiao, Wei Shen. "Rethinking Normalization and Elimination Singularity in Neural Networks." arXiv:1911.09738 (2019). https://pith.science/paper/MINPENK3

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,252 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.14 · human participants 0.02 · controlled comparison 0.67 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.95 · mentions code or data availability 0.98.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

In this paper, we study normalization methods for neural networks from the perspective of elimination singularity. Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated. They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances. We show that channel-based normalizations (e.g. Layer Normalization and Group Normalization) are unable to guarantee a far distance from elimination singularities, in contrast with Batch Normalization which by design avoids models from getting too close to them. To address this issue, we propose BatchChannel Normalization (BCN), which uses batch knowledge to avoid the elimination singularities in the training of channel-normalized models. Unlike Batch Normalization, BCN is able to run in both large-batch and micro-batch training settings. The effectiveness of BCN is verified on many tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is here: https://github.com/joe-siyuan-qiao/Batch-Channel-Normalization.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	In this paper, we study normalization methods for neural networks from the perspective of elimination singularity.	contribution 0.82	0.79	0.03	asserted 0.58
s1	Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated.	background 1.00	0.12	0.03	asserted 1.00
s2	They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances.	background 0.95	0.17	0.05	asserted 0.95
s3	We show that channel-based normalizations (e.g. Layer Normalization and Group Normalization) are unable to guarantee a far distance from elimination singularities, in contrast with Batch Normalization which by design avoids models from getting too close to them.	result 0.85	0.83	0.30	asserted 1.00
s4	To address this issue, we propose BatchChannel Normalization (BCN), which uses batch knowledge to avoid the elimination singularities in the training of channel-normalized models.	contribution 0.61	0.88	0.33	asserted 1.00
s5	Unlike Batch Normalization, BCN is able to run in both large-batch and micro-batch training settings.	contribution 0.78	0.78	0.09	asserted 1.00
s6	The effectiveness of BCN is verified on many tasks, including image classification, object detection, instance segmentation, and semantic segmentation.	result 0.93	0.93	0.20	asserted 1.00
s7	The code is here: https://github.com/joe-siyuan-qiao/Batch-Channel-Normalization.	contribution 0.67	0.21	0.02	unclear 0.67

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

In this paper, we study normalization methods for neural networks from the perspective of elimination singularity. Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated. They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances. We show that channel-based normalizations (e.g. Layer Normalization and Group Normalization) are unable to guarantee a far distance from elimination singularities, in contrast with Batch Normalization which by design avoids models from getting too close to them. To address this issue, we propose BatchChannel Normalization (BCN), which uses batch knowledge to avoid the elimination singularities in the training of channel-normalized models. Unlike Batch Normalization, BCN is able to run in both large-batch and micro-batch training settings. The effectiveness of BCN is verified on many tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is here: https://github.com/joe-siyuan-qiao/Batch-Channel-Normalization.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

5 arXiv papers cite this in 7 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Cites it as background 2 papers

The success of BNs was initially attributed to overcoming internal covariate shift [ref] , but later on raises many open discussions on its effect of improving landscape smoothness [ref] ; enabling larger learning rates [ref] and reducing gradient sensitivity [ref] ; preserving the rank of pre-activation weight matrices [ref] ; decoupling feature length and direction [ref] ; capturing domain-specific artifacts [ref] ; reducing BN's dependency on batch size [ref] ; preventing elimination singularities [ref] ; and even characterizing an important portion of network expressivity [ref] .

Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity · Normalization in Deep Learning · 0.99
Besides, Qiao et al. [ref] introduced another combination that stacked the BN with the GN rather than conducting a weighted summation operation.

A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical images · Normalization · 0.86

What it cites · 10 sentences without a confident label

Uses the method or tool of 2 papers

We choose DeepLabV3 [ref] as the evaluation model for its good performances and its use of the pre-trained ResNet-101 backbone.

Rethinking Atrous Convolution for Semantic Image Segmentation · Semantic Segmentation on PASCAL VOC · 0.91
Please see [ref] for more details.

MMDetection: Open MMLab Detection Toolbox and Benchmark · Object Detection and Segmentation on COCO · 0.83

Compares with 1 paper

Table [ref] shows comparisons with more recent normalization methods, Switchable Normalization (SN) [ref] and Dynamic Normalization (DN) [ref] which were tested for a variant of ResNet for CIFAR: ResNet-18.

Differentiable Learning-to-Normalize via Switchable Normalization · Image Classification on CIFAR · 1.00

Cites as background 6 papers

For example, Batch Normalization (BN) [ref] and Layer Normalization (LN) [ref] were proposed based on the conjecture that they are able to reduce internal covariate shift which negatively impacts training.

Layer Normalization · Introduction · 0.95
Besides them, there are also weight-based normalization methods, such as Weight Normalization [ref] and Weight Standardization [ref] .

Micro-Batch Training with Batch-Channel Normalization and Weight Standardization · Related Work · 0.97
[ref] also explores length-direction decoupling in BN and [ref] .

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization · Related Work · 0.97

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.