REVIEW 3 major objections 5 minor 18 references
Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Hybrid batch norm beats standard BN in federated learning by separating statistics from weights.
desk verdict Useful and credible HBN normalization for FL, but the 'unbiased' claim only holds under full participation and the evaluation protocol needs to be stated clearly. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the sequential update of statistical versus learnable parameters, combined with the pooled-variance identity. In each round a client first evaluates the frozen global model on its own data to get local statistics (Equation 7); the server then forms the global estimate by the exact decomposition (Equation 8), μ_g = Σ (N_k/N) μ_k and (σ²)_g = 1/(N-1) Σ N_k[(σ²)_k + (μ_k − μ_g)²], which recovers the population statistics of the global model if every client reports. The second mechanism is the hybrid distribution factor α, a per-channel scalar that sits in a sigmoid and mixes batch statistics with global statistics (Equation 9), with the mixture learned by gradient descent on each client and never sent to the server; at evaluation time only the global statistics are used.
What would settle it
Run HBN on the same benchmarks with C=1.0 (full participation, no moving average) and compare the server-side aggregated statistics against directly computed statistics of the global model on the union of all client data; if the aggregated statistics deviate measurably or the accuracy advantage vanishes, the unbiasedness claim fails.
Extended reading notes
Core claim
The central claim is that the dilemma of batch normalisation in federated learning is solvable if the statistical parameters (means and variances) are decoupled from the learnable parameters and updated in the right order. Computing each client's statistics with the frozen global model makes them functions of a common model, and the server-side pooling rule reproduces the exact mean and variance of the global model over the union of all client data, up to the assumption that all clients report their statistics. The HBN layer then normalises each batch with a channel-wise learnable mixture of the real-time local batch statistics and these historical global statistics, so a client can trust global structure when its own batch is uninformative. The paper shows that this design outperforms BN, GN, LN, FedFN, FixBN, and FBN across datasets, heterogeneity levels, batch sizes, numbers of clients, and network backbones.
Load-bearing premise
The claim that Equation (8) yields unbiased global statistics assumes that every client's data is fully observed in every round; in the experiments only 10% of clients participate and a moving average (λ=0.01) is applied, so the practical gains rest on the unstated assumption that the sampled clients are representative.
Editorial extensions
If this is right
- On CIFAR-10/100 and Tiny-ImageNet with strong Dirichlet heterogeneity and batch size 4, HBN is the top accuracy across every setting, e.g., 78.22 vs 75.82% for BN on CIFAR-10 (ϕ=0.6).
- HBN is markedly more robust to shrinking batch size: from B=32 to B=2 on CIFAR-100 its accuracy falls 2.19 points versus BN's 5.74, and at B=2 it still beats BN at B=32.
- The gains persist as the client population grows from 100 to 1000 clients, where local-statistics methods degrade sharply.
- HBN acts as a plugin for FedAvg, FedProx, FedAdam, Scaffold, Moon, FedSAM, FedACG, and Fedwon, improving each one.
- HBN also lifts accuracy in MobileNetV2, ResNet-18/50, and VGG-11/19 compared with BN and GN.
Reading between the lines
- The unbiasedness guarantee in Equation (8) is derived for full participation; the experiments use C=0.1 with a moving average (λ=0.01), so the practical gains may come from smoothing partial-participation noise rather than from exact unbiasedness. A test that varies C while holding λ fixed would separate these explanations.
- The evaluation protocol of Section 4.1 computes test accuracy with statistics collected from the same client subset as the final round; a fairer probe would evaluate with global statistics aggregated from all clients, or with a separate server-side validation set.
- The hybrid factor α can be read as a per-channel measure of how much each client trusts the global distribution; a natural extension is to analyse or regularise α to detect distribution shift and client drift.
- The same sequential-statistics idea could transfer to other distributed settings, such as split learning or asynchronous FL, where the statistics are also computed on stale models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hybrid Batch Normalisation (HBN), a normalization layer for federated learning. In each communication round, clients first compute local BN statistics from the downloaded global model before any local training (Eq. (7)); the server then aggregates these statistics with the weighted formula in Eq. (8), which the authors prove in Appendix A.2 gives the exact global statistics of the previous global model when all clients participate. During local training, HBN normalizes each mini-batch by mixing current batch statistics with the stored global statistics through a per-client, per-channel learnable factor α (Eq. (9)). The method is evaluated on CIFAR-10/100 and Tiny-ImageNet with several baselines and is shown to improve over BN, GN, LN, FedFN, FixBN and FBN, especially for small batches and heterogeneous data. The paper also reports compatibility with several advanced FL optimizers and classic network architectures.
Significance. If the empirical claims hold, HBN is a practically useful and easy-to-integrate normalization layer for federated learning, addressing a real limitation of BN under non-IID data and small batch sizes. The paper has clear strengths: it releases code, provides a self-contained derivation of the unbiased global statistics identity in Appendix A.2, and runs extensive experiments across datasets, heterogeneity levels, batch sizes, client numbers, architectures, and FL optimizers. The central idea of separating the update of statistical parameters from learnable parameters is well motivated. However, the reported experiments operate in a regime where the exact unbiased-statistics guarantee does not directly apply, and the evaluation protocol is underspecified in a way that could affect the fairness of the comparisons. These issues need to be addressed before the central claims can be accepted.
major comments (3)
- [Section 3.2 / Appendix A.2 / Section 4.1] The unbiased-statistics guarantee in Eq. (8) is derived under full participation: every client k ∈ [K] contributes its local statistics computed on its complete dataset D_k. The experiments, however, use C = 0.1 (10 of 100 clients per round) and replace Eq. (8) with the exponentially moving average in Eq. (11) using λ = 0.01. In the reported setup, the stored global statistics are therefore not the unbiased estimates of S(ω_g^{t-1}; D_g) that the paper's central contribution claims; they are a heavily smoothed aggregate over a small client subset. The paper should either (a) add experiments with full participation (C = 1, λ = 1) that directly exercise the unbiased mechanism, or (b) re-frame the headline claim and provide a bias analysis for partial participation. As written, the theoretical guarantee and the empirical regime do not match.
- [Section 4.1] The evaluation protocol is underspecified: the sentence "we test the statistical parameters of the same client subset for accuracy evaluation" does not state whether the test set covers all clients or only those clients selected in the last communication round. If the latter, HBN is evaluated on the same clients used to compute its final statistics, which can inflate its advantage over baselines whose statistics are evaluated on all clients. The authors must state explicitly which test distribution is used and, if only a subset is used, rerun the evaluation on the full test set or on held-out clients before the claim that HBN consistently outperforms existing normalization solutions can be accepted.
- [Section 4.3, Table 6] Ablation row (2), labeled "Equation (8) ← weighted average", is ambiguous because Eq. (8) is itself a weighted average with a between-client correction term. The description in the text, "summarise local statistics based on weighted average", does not identify what replacement is actually implemented. If the row replaces Eq. (8) with the naive weighted average of local statistics from Eq. (6), this should be stated explicitly; if not, the reader cannot tell which design choice is being ablated. This matters because the row is used to argue that the unbiased aggregation formula is a load-bearing component.
minor comments (5)
- [Section 4.3 / Table 6] In the ablation description, "moving average without separation" for row (1) is clear enough, but the distinction between rows (1) and (2) should be made precise in terms of which quantities are uploaded and how they are aggregated, so that the reader can reproduce the ablation configuration.
- [Appendix A.1, Eq. (14)] The chain of equalities in Eq. (14) uses |Σ a_i| = Σ |a_i|, which is not an equality in general; it should be an inequality (triangle inequality). The qualitative conclusion about bias remains, but the displayed derivation is technically incorrect.
- [Section 4.3 / Table 3] The notation for the Dirichlet coefficient is inconsistent: the text uses ϕ throughout most of the paper, but Table 3 and Appendix B.3 use β. Please unify the notation.
- [Table 1] The Tiny-ImageNet results for FBN (4.90 and 3.76) are far below all other methods, including GN and LN; this suggests the learning-rate range may not have covered a workable setting for FBN, and the paper does not discuss this. A brief note or an extended search would strengthen the comparison.
- [Section 4.2 / Tables 1-5] The paper reports only point estimates of test accuracy without error bars or repeated-run standard deviations. Given that the reported gains are often 1-3 percentage points, reporting variance across seeds would help assess whether the improvements are significant.
Circularity Check
No circularity: HBN's unbiased-statistics claim is an algebraic consequence of the frozen-model protocol, not a reduction to its own inputs.
full rationale
The central theoretical claim is that the aggregation rule in Eq. (8) yields unbiased global statistics S(omega_g^{t-1}; D_g). The paper proves this in Appendix A.2: when each client computes its local statistics on its full local dataset with the previous global model frozen (Eqs. 7 and 15), the weighted aggregation is exactly the mean and variance of that global model over the union of all client data (Eq. 16). This is an identity derived from the protocol's definitions, not a fitted parameter relabeled as a prediction, and it depends on no self-citation. The learnable hybrid factor alpha is trained on the training objective and ablated in Table 6, so it is not assumed into existence. The only overlapping self-citation (Zhong et al., 2023, whose author list includes H.-Y. Chen) is used to describe the FixBN baseline and is not load-bearing for HBN's contributions. The paper does contain a genuine empirical limitation: the unbiasedness identity requires full participation and full local data, while the experiments activate C=0.1 clients and replace Eq. (8) with the moving-average update in Eq. (11); Section 4.1 also states that evaluation uses the statistics of the same client subset. These are validity and reproducibility concerns about the experimental protocol, not circular reasoning in the derivation chain. Accordingly, no circular step is exhibited, and the score is 0.
Assumptions & free parameters
free parameters (2)
- hybrid distribution factor alpha (per channel, per client)
- moving-average momentum lambda for global statistics =
0.01
assumptions (3)
- domain assumption All clients participate in every round when computing global statistics
- ad hoc to paper The global statistics of the previous round's model remain useful for normalizing the current round's training batches
- domain assumption Client data partitions are fixed and each client's data distribution is stable across rounds
Cite this review
Pith. "Pith review of Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning." pith.science (2026). https://pith.science/paper/7OYLWAR7
@misc{pith2026250521877,
author = {Pith},
title = {Pith review of: Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7OYLWAR7}},
note = {Machine review of arXiv:2505.21877}
}
read the original abstract
Batch Normalisation (BN) is widely used in conventional deep neural network training to harmonise the input-output distributions for each batch of data. However, federated learning, a distributed learning paradigm, faces the challenge of dealing with non-independent and identically distributed data among the client nodes. Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance. To this end, it is urgent to explore an alternative normalisation solution for federated learning. In this work, we resolve the dilemma of the BN layer in federated learning by developing a customised normalisation approach, Hybrid Batch Normalisation (HBN). HBN separates the update of statistical parameters (i.e. , means and variances used for evaluation) from that of learnable parameters (i.e. , parameters that require gradient updates), obtaining unbiased estimates of global statistical parameters in distributed scenarios. In contrast with the existing solutions, we emphasise the supportive power of global statistics for federated learning. The HBN layer introduces a learnable hybrid distribution factor, allowing each computing node to adaptively mix the statistical parameters of the current batch with the global statistics. Our HBN can serve as a powerful plugin to advance federated learning performance. It reflects promising merits across a wide range of federated learning settings, especially for small batch sizes and heterogeneous data.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Each client k ∈ [K] downloads the global model as the initialisation of the local model, and trains it on local dataset Dk. Due to the fact that the statistical parameters in vanilla BN are updated synchronously with the learnable parameters, the statistical parameters obtained in each locality are approximate estimates of ωt k = ωt−1 g − η∇L(ωt−1 g , Dk)...
work page 2024
-
[5]
Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual clas- sification. arXiv preprint arXiv:1909.06335,
arXiv 1909
-
[6]
Batch normalization: Accelerating deep net- work training by reducing internal covariate shift
Ioffe, S. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167,
-
[9]
On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,
Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,
arXiv 1907
-
[10]
Fedbn: Federated learning on non-iid features via local batch normalization
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021b. Lubana, E. S., Dick, R., and Tanaka, H. Beyond batchnorm: Towards a unified understanding of normalization in deep learning. NeurIPS, 34:4778–4791,
-
[12]
Continual normalization: Rethinking batch normalization for online continual learn- ing
Pham, Q., Liu, C., and Hoi, S. Continual normalization: Rethinking batch normalization for online continual learn- ing. arXiv preprint arXiv:2203.16102,
-
[15]
Xu, J., Wang, S., Wang, L., and Yao, A. C.-C. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874,
-
[16]
Federated learning with non-iid data.arXiv preprint arXiv:1806.00582,
Zhao, Y ., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V . Federated learning with non-iid data.arXiv preprint arXiv:1806.00582,
Show all 18 references
-
[2009]
One weird trick for parallelizing convolu- tional neural networks
Krizhevsky, A. One weird trick for parallelizing convolu- tional neural networks. arXiv preprint arXiv:1404.5997,
-
[2015]
BN can effectively solve this problem under centralised training
refers to the change in the input distribution of each layer during the training of deep neural networks due to updates in model parameters. BN can effectively solve this problem under centralised training. However, in the context of federated learning, the changes in model pa...
2022
-
[2017]
arXiv preprint arXiv:1706.02677,
-
[2018]
and Zisserman, A
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,
-
[2019]
Overcoming the challenges of batch normalization in federated learning
Guerraoui, R., Pinot, R., Rizk, G., Stephan, J., and Taiani, F. Overcoming the challenges of batch normalization in federated learning. arXiv preprint arXiv:2405.14670,
-
[2020]
Ba, J. L. Layer normalization. arXiv preprint arXiv:1607.06450,
-
[2021]
Towards un- derstanding regularization in batch normalization
Luo, P., Wang, X., Shao, W., and Peng, Z. Towards un- derstanding regularization in batch normalization. arXiv preprint arXiv:1809.00846,
-
[2022]
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Koneˇcn`y, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arXiv preprint arXiv:2003.00295,
2003 arXiv
-
[2023]
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[2024]
A downsam- pled variant of imagenet as an alternative to the cifar datasets
Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of imagenet as an alternative to the cifar datasets. ArXiv, abs/1707.08819,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.