U-Net Training with Instance-Layer Normalization

Guang-Zhong Yang; Peichao Li; Xiao-Yun Zhou; Zhao-Yang Wang

REVIEW 4 major objections 6 minor 18 references

U-Net Training with Instance-Layer Normalization

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a trainable blend of Instance Normalization and Layer Normalization, followed by Group Normalization, gives a U-Net a consistent accuracy edge in cardiac MRI segmentation, with reported Dice gains of almost 2% over…

desk verdict Plausible normalization variant, but the ~2% Dice claim rests on single two-epoch runs with learning-rate selection after the fact; the evidence is too weak to support it. read the letter →

arxiv 1908.08466 v2 pith:ME52KJKQ submitted 2019-08-21 eess.IV cs.CVcs.LG

Xiao-Yun Zhou , Peichao Li , Zhao-Yang Wang , Guang-Zhong Yang This is my paper · ORCID

classification eess.IVcs.CVcs.LG

keywords Instance-LayerNormalizationU-NetcardiacMRIsegmentationrightventricleleftgroupDicesimilaritycoefficientbiomedicalimage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Every convolutional neural network used for medical image segmentation inserts normalization layers that rescale intermediate feature maps for stable training. This paper argues that the normalization strategy need not be fixed per layer, and proposes Instance-Layer Normalization (ILN): normalize the same feature map with instance statistics and with layer statistics, blend the two normalized versions with a trainable weight passed through a sigmoid, and then apply group normalization with 16 groups to restore a normalized distribution. On U-Net segmentation of the right and left ventricles in cardiac MRI, the paper reports that ILN achieves higher Dice scores than no normalization, Instance Normalization, Layer Normalization, and Group Normalization in five of the six cross-validation folds, with gains described as almost 2% Dice. The authors present ILN as a simple drop-in layer whose two design choices—a differentiable blend and a renormalization cascade—fix known problems in Batch-Instance Normalization.

What carries the argument

The load-bearing object is the ILN layer itself, a composition of three standard operations: a per-sample instance normalization over $(H,W)$, a per-sample layer normalization over $(H,W,C)$, and a group normalization with 16 groups applied after their convex combination. The blend weight is $\rho$ transformed by the sigmoid, so the layer can interpolate continuously between instance and layer statistics and the gradient never hits a non-differentiable corner. The added GN16 stage is what restores zero-mean, unit-variance statistics to the blended feature map, which the paper argues is necessary for signal propagation in deep networks.

What would settle it

Re-run the same U-Net and the same RV/LV folds with each normalization method (None, IN, LN, GN4, ILN) under many random initializations and a fixed learning-rate schedule, then compare Dice distributions. If ILN's advantage over the best baseline falls within the inter-run spread, the central claim of consistent improvement is not supported.

Watch

Extended reading notes

Core claim

The paper's central discovery is that combining Instance Normalization and Layer Normalization with a smooth, trainable weight, and then renormalizing the combination, is better than using any one of the standard normalization schemes alone. Formally, for a feature map $F$ of shape $(N,H,W,C)$, ILN computes instance-normalized $\hat{F}^I$ and layer-normalized $\hat{F}^L$, forms $\hat{F}^{IL}=\sigma(\rho)\hat{F}^I+(1-\sigma(\rho))\hat{F}^L$ where $\sigma$ is the sigmoid and $\rho$ is a trainable scalar, then applies Group Normalization with 16 channel groups to $\hat{F}^{IL}$ and finishes with per-channel affine parameters $\gamma,\beta$. The authors claim this fixes two defects of Batch-Instance Normalization: the Clip function's non-differentiability at 0 and 1, and the loss of a normalized distribution after blending. Their experiments on 6082 right-ventricle images and 805 left-ventricle images with a U-Net show ILN ahead of the baselines in most folds, which they summarize as noticeable accuracy improvements of almost 2% DSC.

Load-bearing premise

The largest assumption is that the 0.005–0.025 Dice differences between normalization methods are real method effects rather than training noise, because each method was evaluated with a single two-epoch run per fold and the reported standard deviations are several times larger than those differences.

Editorial extensions

If this is right

A correct ILN would give medical-image segmentation a drop-in normalization layer: no new loss term, no architectural change, and only one extra trainable scalar per layer plus a group-normalization pass.
On cardiac MRI, gains of almost 2% Dice would translate into better ventricle boundary overlap in a clinical imaging pipeline at negligible additional cost.
Because the blend is differentiable everywhere, gradient-based training can smoothly adapt the normalization regime layer by layer, which the paper shows through layer-specific $\rho$ training curves.
The GN16 cascade suggests a general recipe: whenever a normalization method produces a combined feature map that is no longer normalized, a subsequent normalization stage can repair the distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

I would not treat the claimed ranking as stable until multi-seed runs confirm it: Table 3's standard deviations (about 0.10–0.32 Dice) are several times larger than the mean differences between methods (0.005–0.025), and each method was run once for two epochs per fold.
A natural extension is to apply the same sigmoid-blend-plus-renormalize recipe to other normalization pairs, such as Batch plus Instance or Group plus Layer, and to other network families; the paper frames ILN as generic but validates it only on U-Net segmentation.
The reported advantage may be partly an optimization effect—the smooth blend and renormalization may change the effective learning-rate landscape—rather than a property of the normalized statistics themselves; fixing learning-rate selection across methods would separate these.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a clearly written, modest empirical paper that proposes ILN, a normalization layer which mixes Instance Normalization and Layer Normalization via a sigmoid-weighted sum and then applies Group Normalization with 16 groups. The specific design is new and the motivation is sensible: it fixes two known issues in Batch-Instance Normalization (non-differentiable clip, and loss of normalized distribution after combining). The authors also take care to re-implement all baselines in the same low-level TensorFlow framework and to report overhead. Credit where due: the method is simple, plausible, and the paper is honest about the one validation where it does not help.

The problem is the evidence. Table 3 shows ILN ahead of IN, LN, GN4 and no-normalization by 0.5–2.5 Dice points, but the per-subject standard deviations are 0.098–0.323, an order of magnitude larger. Each cell is a single run with only two training epochs, and the learning rate was chosen per method after seeing results. That protocol makes the reported numbers upper bounds and the ranking could easily flip with a different seed or split. There are no repeated runs and no significance tests. So the headline claim of 'noticeable accuracy improvements' is not supported by the data. I also note that the original BIN baseline is never tested in the main comparison; Table 1 compares clip vs sigmoid vs softmax for combining IN/LN, which is not the same as BIN's BN/IN combination. No code or data are released, which hurts reproducibility.

The paper itself is not a bad paper. The idea is reasonable, the write-up is clear, and the authors are candid about limitations (they explicitly acknowledge the LV-3 failure and say the ground-truth rho is unknown). The weaknesses are in the evaluation, not in the reasoning. With repeated seeds, longer training, significance tests, and ideally a released implementation, the claims could be tested properly. As it stands, I'd treat the 1–2% Dice improvement as unverified.

For peer review: I'd send this to a serious referee. The topic is relevant, the method is a reasonable incremental contribution, and the shortcomings are fixable. But I would not cite it in my own work yet, and I wouldn't bring it to reading group as a strong result—maybe as an example of how easy it is to over-claim from underpowered experiments.

Referee Report

4 major / 6 minor

Summary. The paper proposes Instance-Layer Normalization (ILN), which combines Instance Normalization and Layer Normalization feature maps using a Sigmoid-weighted trainable parameter, and then applies Group Normalization with 16 groups (GN16) to the combined feature map. The method is motivated by two perceived weaknesses of Batch-Instance Normalization: the non-differentiability of the Clip function at 0 and 1, and the loss of a normalized distribution after combining feature maps. The authors validate ILN on right-ventricle and left-ventricle cardiac MRI segmentation with a U-Net, comparing against no normalization, Instance Normalization, Layer Normalization, and Group Normalization over three-fold cross-validation. They report that ILN achieves the highest Dice Similarity Coefficient in five of six folds and claim an improvement of almost 2% DSC, concluding that ILN is an effective normalization strategy for medical image segmentation.

Significance. If the claimed accuracy gains were robust, ILN would be a useful and inexpensive drop-in normalization layer: it adds only one trainable parameter per layer plus a group-normalization step, and the manuscript tests it on public cardiac MRI benchmarks. A clear strength is that the authors re-implement all baselines in a common low-level TensorFlow framework, which is fairer than comparing against high-level API implementations. The paper is clearly written and the architectural idea is plausible. However, the current evidence is not strong enough to establish the central claim: the evaluation uses single two-epoch runs, post-hoc learning-rate selection, and mean differences between methods that are an order of magnitude smaller than the reported per-subject standard deviations. The authors honestly acknowledge one failure (LV-3) and that generalization to other tasks needs further validation, but the quantitative conclusion requires substantially stronger experimental support before the claimed improvement can be considered established.

major comments (4)

[Section 2.2 and Table 3] The central claim that ILN outperforms IN, LN, and GN4 rests on a single two-epoch run per method with the best learning rate selected post hoc. The mean ILN-minus-IN differences in Table 3 are 0.005, 0.022, 0.025, 0.014, 0.017, and -0.009 across RV-1/2/3 and LV-1/2/3, while the reported per-subject standard deviations range from 0.098 to 0.323. Because no repeated-seed variance is reported and no significance test is applied, the observed differences are within the noise floor and do not establish a consistent improvement.
[Section 2.2] Selecting the best of five initial learning rates from the test-fold results is a form of selection bias, making the reported values upper bounds rather than unbiased estimates of expected performance. The authors should either fix the learning rate a priori, tune it on a held-out validation split, or report the performance for all five learning rates and multiple random seeds.
[Sections 2.2 and 5] Training for only two epochs, with the learning rate divided by five at the second epoch, is an arbitrary early stopping point; rankings at epoch two may not reflect converged or practically relevant performance. The 'almost 2% DSC' claim in Section 5 is also not tied to a specific baseline: the mean ILN advantage over IN is about 1.2%, whereas over LN and GN4 it is about 2.3-2.8%, so the claimed improvement depends on the chosen comparison.
[Section 3.1 and Table 1] The experiment intended to validate Sigmoid over Clip does not include the original BIN method: the Clip row combines IN and LN with a single clipped weight, whereas BIN combines BN and IN. Consequently, the paper does not directly demonstrate an improvement over the actual BIN method it criticizes in the introduction.

minor comments (6)

[Equations (5) and (6)] The text states that '// is exact division', but in common programming languages '//' denotes floor division; if the channel count C is not a multiple of 16, the grouping formula with M = C//16 leaves some channels unnormalized or misassigned. Please clarify the intended grouping for arbitrary C.
[Table 3] Since ILN contains GN16 as a component, the main comparison table should include a GN16-alone row rather than relying on the statement that GN16 'performed similarly to IN [16]'; without this row, the reader cannot separate the contribution of the Sigmoid combination from the contribution of the added GN16 step.
[Section 3.5] The training curves of ρ in Figure 3 are presented without quantitative analysis, and the text acknowledges that the correctness of the curves cannot be judged; consider reporting the distribution or final values of ρ across layers, or relating them to layer depth, to make the figure informative.
[Section 1] The claim that the Clip function 'is not differentiable at input values of 0 and 1' is true only at two isolated points, and subgradients are standard in deep learning; the paper should state why this non-differentiability is practically harmful rather than merely theoretically present.
[Section 2.2] Several experimental details are missing, including the batch size, the number of iterations per epoch, the exact U-Net configuration (e.g., number of layers and channels at each depth), and any preprocessing steps beyond rotation augmentation; these details are needed for reproducibility.
[Table 1] The Softmax row uses two trainable parameters while the Clip and Sigmoid rows use one, so the comparison is not parameter-matched; please discuss whether the extra parameter affects the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: ILN's performance is an empirical result from held-out validation folds, not derived from its own definition.

full rationale

No circular derivation chain is present. ILN is defined independently in Section 2.1: the combined feature map is constructed by Eq. (4) as a Sigmoid-weighted sum of IN and LN outputs, and then normalized by GN16 in Eqs. (5) and (6). The claimed accuracy improvement is obtained empirically from three-fold cross-validation on held-out test groups, with results reported in Table 3; there is no equation by which the outcome is encoded into the method definition. The ablations in Tables 1 and 2 (Sigmoid vs. Clip vs. Softmax, and with vs. without GN16) are empirical comparisons rather than definitions. The paper cites the authors' own prior work [16] and [18] for the training schedule and for the statement that GN16 performed similarly to IN [16], but these citations only influence experimental protocol and baseline selection; they do not force the rankings in Table 3. Omitting GN16 as a separate baseline based on a self-citation is a design choice, not a load-bearing circular step. The concern raised by the evaluation protocol—single two-epoch runs per method with reported per-subject standard deviations (0.098 to 0.323) several times larger than the mean differences between methods (0.005 to 0.025)—is a statistical validity concern, not circularity. The method is self-contained against external benchmarks, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper contributes a novel normalization layer with one trainable interpolation parameter per layer plus a fixed group-normalization block; no new physical entities are introduced. The main hand choices are the group count, training length, and learning-rate selection, which together make the empirical comparison fragile.

free parameters (4)

GN group count = 16 (chosen by hand)
The group number for the post-combination group normalization is fixed at 16 without a sweep over group sizes; only with and without GN16 is tested.
Number of training epochs = 2
Training is limited to two epochs following [16,18]; this short schedule may favor certain normalization methods and limits the generality of conclusions.
Learning rate per experiment = best of 5 values (1.5, 1.0, 0.5, 0.1, 0.05)
Each method reports the best of five initial learning rates, so results are conditional on this selection and may reflect quick convergence under favorable hyperparameters.
Rho initialization = 0.5
The interpolation parameter is initialized at 0.5, which through sigmoid gives initial weight 0.622 for IN and 0.378 for LN; the effect of this choice is not explored.

assumptions (3)

domain assumption The ground-truth segmentations and the Dice Similarity Coefficient are valid measures of segmentation quality for the RV and LV tasks.
All conclusions are based on DSC computed against expert labels; if labels or the metric are flawed, the ranking of methods may change. This is assumed throughout Section 3.
domain assumption A feature map with a normalized distribution after combination is beneficial for signal propagation in DCNNs.
This premise motivates adding GN16 in Section 2.1; it is tested only indirectly by comparing with and without GN16.
domain assumption The two-epoch training protocol and the selected hyperparameters are sufficient to determine the relative merit of normalization layers.
The paper uses a short training schedule and single runs to rank methods, which assumes that observed differences reflect method quality rather than training noise (Section 2.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of U-Net Training with Instance-Layer Normalization." pith.science (2026). https://pith.science/paper/ME52KJKQ

@misc{pith2026190808466,
  author       = {Pith},
  title        = {Pith review of: U-Net Training with Instance-Layer Normalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ME52KJKQ}},
  note         = {Machine review of arXiv:1908.08466}
}

read the original abstract

Normalization layers are essential in a Deep Convolutional Neural Network (DCNN). Various normalization methods have been proposed. The statistics used to normalize the feature maps can be computed at batch, channel, or instance level. However, in most of existing methods, the normalization for each layer is fixed. Batch-Instance Normalization (BIN) is one of the first proposed methods that combines two different normalization methods and achieve diverse normalization for different layers. However, two potential issues exist in BIN: first, the Clip function is not differentiable at input values of 0 and 1; second, the combined feature map is not with a normalized distribution which is harmful for signal propagation in DCNN. In this paper, an Instance-Layer Normalization (ILN) layer is proposed by using the Sigmoid function for the feature map combination, and cascading group normalization. The performance of ILN is validated on image segmentation of the Right Ventricle (RV) and Left Ventricle (LV) using U-Net as the network architecture. The results show that the proposed ILN outperforms previous traditional and popular normalization methods with noticeable accuracy improvements for most validations, supporting the effectiveness of the proposed ILN.

Figures

Figures reproduced from arXiv: 1908.08466 by the authors.

**Figure 1.** The curves of Clip and Sigmoid function. However, Clip function is not differentiable at input values of 0 and 1. In this paper, Sigmoid function Sigmoid(x) = 1/(e −x + 1) which is differentiable everywhere is applied to solve this potential issue: Fˆ IL = Sigmoid(ρ) · Fˆ I + (1 − Sigmoid(ρ)) · Fˆ L (4) An additional potential issue in the original BIN is that the combined Fˆ IL is no longer with a mean of 0 and a … view at source ↗

**Figure 2.** Eight examples were selected randomly from the RV and LV segmentation results, where red indicates the ground truth, green indicates the segmentation result, and yellow indicates the true positives of the prediction. 3.4 Comparison to Other Methods The mean±std segmentation DSCs of using no normalization, IN, LN, GN4, and the proposed ILN with the U-Net framework are shown in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** The training curves of eight ρ selected randomly from the 22 layers in U-Net. The CPU used is Intel Xeon(R) E5-1650 v4@3.60GHz×12. The GPU used is Nvidia Titan XP. Comparing ILN to IN, the parameter number increases 22, as one parameter is added to each layer. The training time for 200 iterations increases from 34.8s to 36.5s due to the additional GN16 calculation. 4 Discussion The proposed ILN strategy is generic a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

[1]

Stat 1050, 21 (2016)

Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. Stat 1050, 21 (2016)

work page 2016
[2]

In: NeurIPS

Bjorck, N., Gomes, C.P., Selman, B., Weinberger, K.Q.: Understanding batch nor- malization. In: NeurIPS. pp. 7705–7716 (2018)

work page 2018
[3]

In: NeurIPS

Ioﬀe, S.: Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. In: NeurIPS. pp. 1945–1953 (2017)

work page 2017
[4]

In: ICML

Ioﬀe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. pp. 448–456 (2015)

work page 2015
[5]

In: NeurIPS

Nam, H., Kim, H.E.: Batch-instance normalization for adaptively style-invariant neural networks. In: NeurIPS. pp. 2563–2572 (2018)

work page 2018
[6]

The MIDAS Journal- Cardiac MR Left Ventricle Segmentation Challenge 49 (2009)

Radau, P., Lu, Y., Connelly, K., Paul, G., Dick, A., Wright, G.: Evaluation frame- work for algorithms segmenting short axis cardiac MRI. The MIDAS Journal- Cardiac MR Left Ventricle Segmentation Challenge 49 (2009)

work page 2009
[7]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241 (2015)

2015
[8]

In: NeurIPS

Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In: NeurIPS. pp. 901–909 (2016)

work page 2016

Show all 18 references

[9]

Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? In: NeurIPS. pp. 2488–2498 (2018)

2018
[10]

arXiv preprint arXiv:1607.08022 (2016)

Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)

2016 arXiv
[11]

arXiv preprint arXiv:1802.03133 (2018)

Wang, G., Peng, J., Luo, P., Wang, X., Lin, L.: Batch kalman normaliza- tion: Towards training deep neural networks with micro-batches. arXiv preprint arXiv:1802.03133 (2018)

2018 arXiv
[12]

In: ECCV

Wu, Y., He, K.: Group normalization. In: ECCV. pp. 3–19 (2018)

2018
[13]

In: NeurIPS

Xu, Y., Wang, X.: Understanding weight normalized deep neural networks with rectiﬁed linear units. In: NeurIPS. pp. 130–139 (2018)

2018
[14]

IEEE RAL 3(2), 1314–1321 (2018)

Zhou, X.Y., Lin, J., Riga, C., Yang, G.Z., Lee, S.L.: Real-time 3D shape instan- tiation from single ﬂuoroscopy projection for fenestrated stent graft deployment. IEEE RAL 3(2), 1314–1321 (2018)

2018
[15]

In: 2018 IEEE/RSJ IROS

Zhou, X.Y., Riga, C., Lee, S.L., Yang, G.Z.: Towards automatic 3D shape instan- tiation for deployed stent grafts: 2D multiple-class and class-imbalance marker segmentation with equally-weighted focal U-Net. In: 2018 IEEE/RSJ IROS. pp. 1261–1267 (2018)

2018
[16]

IEEE RAL (2019)

Zhou, X.Y., Yang, G.Z.: Normalization in training U-Net for 2D biomedical se- mantic segmentation. IEEE RAL (2019)

2019
[17]

MedIA 44, 86–97 (2018)

Zhou, X.Y., Yang, G.Z., Lee, S.L.: A real-time and registration-free framework for dynamic shape instantiation. MedIA 44, 86–97 (2018)

2018
[18]

arXiv preprint arXiv:1901.09203 (2019)

Zhou, X.Y., Zheng, J.Q., Yang, G.Z.: Atrous convolutional neural network (ACNN) for biomedical semantic segmentation with dimensionally lossless feature maps. arXiv preprint arXiv:1901.09203 (2019)

2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Stat 1050, 21 (2016)

Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. Stat 1050, 21 (2016)

work page 2016

[2] [2]

In: NeurIPS

Bjorck, N., Gomes, C.P., Selman, B., Weinberger, K.Q.: Understanding batch nor- malization. In: NeurIPS. pp. 7705–7716 (2018)

work page 2018

[3] [3]

In: NeurIPS

Ioﬀe, S.: Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. In: NeurIPS. pp. 1945–1953 (2017)

work page 2017

[4] [4]

In: ICML

Ioﬀe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. pp. 448–456 (2015)

work page 2015

[5] [5]

In: NeurIPS

Nam, H., Kim, H.E.: Batch-instance normalization for adaptively style-invariant neural networks. In: NeurIPS. pp. 2563–2572 (2018)

work page 2018

[6] [6]

The MIDAS Journal- Cardiac MR Left Ventricle Segmentation Challenge 49 (2009)

Radau, P., Lu, Y., Connelly, K., Paul, G., Dick, A., Wright, G.: Evaluation frame- work for algorithms segmenting short axis cardiac MRI. The MIDAS Journal- Cardiac MR Left Ventricle Segmentation Challenge 49 (2009)

work page 2009

[7] [7]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241 (2015)

2015

[8] [8]

In: NeurIPS

Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In: NeurIPS. pp. 901–909 (2016)

work page 2016