Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Hybrid batch norm beats standard BN in federated learning by separating statistics from weights.

desk verdict Useful and credible HBN normalization for FL, but the 'unbiased' claim only holds under full participation and the evaluation protocol needs to be stated clearly. read the letter →

arxiv 2505.21877 v1 pith:7OYLWAR7 submitted 2025-05-28 cs.LG cs.DC

classification cs.LGcs.DC
keywords federatedlearningbatchnormalizationnon-IIDdataglobalstatisticshybriddistributionfactorstatisticalparameterssmallsizeclientheterogeneity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Batch normalisation (BN) breaks down in federated learning because clients update the normalisation statistics and the network weights together, so the averaged local statistics no longer describe the global model. This paper proposes Hybrid Batch Normalisation (HBN), which separates the two updates: each client first computes local statistics with a frozen copy of the global model, the server blends them with a pooled-variance formula to get an unbiased estimate of the global statistics, and each client then trains while mixing its current batch statistics with those global statistics through a learnable factor. The authors claim HBN consistently beats existing normalisation schemes for federated learning, with the largest gains under strong data heterogeneity and tiny batch sizes, and that it works as a drop-in layer for many federated optimisation methods and network architectures.

What carries the argument

The load-bearing mechanism is the sequential update of statistical versus learnable parameters, combined with the pooled-variance identity. In each round a client first evaluates the frozen global model on its own data to get local statistics (Equation 7); the server then forms the global estimate by the exact decomposition (Equation 8), μ_g = Σ (N_k/N) μ_k and (σ²)_g = 1/(N-1) Σ N_k[(σ²)_k + (μ_k − μ_g)²], which recovers the population statistics of the global model if every client reports. The second mechanism is the hybrid distribution factor α, a per-channel scalar that sits in a sigmoid and mixes batch statistics with global statistics (Equation 9), with the mixture learned by gradient descent on each client and never sent to the server; at evaluation time only the global statistics are used.

What would settle it

Run HBN on the same benchmarks with C=1.0 (full participation, no moving average) and compare the server-side aggregated statistics against directly computed statistics of the global model on the union of all client data; if the aggregated statistics deviate measurably or the accuracy advantage vanishes, the unbiasedness claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the dilemma of batch normalisation in federated learning is solvable if the statistical parameters (means and variances) are decoupled from the learnable parameters and updated in the right order. Computing each client's statistics with the frozen global model makes them functions of a common model, and the server-side pooling rule reproduces the exact mean and variance of the global model over the union of all client data, up to the assumption that all clients report their statistics. The HBN layer then normalises each batch with a channel-wise learnable mixture of the real-time local batch statistics and these historical global statistics, so a client can trust global structure when its own batch is uninformative. The paper shows that this design outperforms BN, GN, LN, FedFN, FixBN, and FBN across datasets, heterogeneity levels, batch sizes, numbers of clients, and network backbones.

Load-bearing premise

The claim that Equation (8) yields unbiased global statistics assumes that every client's data is fully observed in every round; in the experiments only 10% of clients participate and a moving average (λ=0.01) is applied, so the practical gains rest on the unstated assumption that the sampled clients are representative.

Editorial extensions

If this is right

  • On CIFAR-10/100 and Tiny-ImageNet with strong Dirichlet heterogeneity and batch size 4, HBN is the top accuracy across every setting, e.g., 78.22 vs 75.82% for BN on CIFAR-10 (ϕ=0.6).
  • HBN is markedly more robust to shrinking batch size: from B=32 to B=2 on CIFAR-100 its accuracy falls 2.19 points versus BN's 5.74, and at B=2 it still beats BN at B=32.
  • The gains persist as the client population grows from 100 to 1000 clients, where local-statistics methods degrade sharply.
  • HBN acts as a plugin for FedAvg, FedProx, FedAdam, Scaffold, Moon, FedSAM, FedACG, and Fedwon, improving each one.
  • HBN also lifts accuracy in MobileNetV2, ResNet-18/50, and VGG-11/19 compared with BN and GN.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unbiasedness guarantee in Equation (8) is derived for full participation; the experiments use C=0.1 with a moving average (λ=0.01), so the practical gains may come from smoothing partial-participation noise rather than from exact unbiasedness. A test that varies C while holding λ fixed would separate these explanations.
  • The evaluation protocol of Section 4.1 computes test accuracy with statistics collected from the same client subset as the final round; a fairer probe would evaluate with global statistics aggregated from all clients, or with a separate server-side validation set.
  • The hybrid factor α can be read as a per-channel measure of how much each client trusts the global distribution; a natural extension is to analyse or regularise α to detect distribution shift and client drift.
  • The same sequential-statistics idea could transfer to other distributed settings, such as split learning or asynchronous FL, where the statistics are also computed on stale models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Hybrid Batch Normalisation (HBN), a normalization layer for federated learning. In each communication round, clients first compute local BN statistics from the downloaded global model before any local training (Eq. (7)); the server then aggregates these statistics with the weighted formula in Eq. (8), which the authors prove in Appendix A.2 gives the exact global statistics of the previous global model when all clients participate. During local training, HBN normalizes each mini-batch by mixing current batch statistics with the stored global statistics through a per-client, per-channel learnable factor α (Eq. (9)). The method is evaluated on CIFAR-10/100 and Tiny-ImageNet with several baselines and is shown to improve over BN, GN, LN, FedFN, FixBN and FBN, especially for small batches and heterogeneous data. The paper also reports compatibility with several advanced FL optimizers and classic network architectures.

Significance. If the empirical claims hold, HBN is a practically useful and easy-to-integrate normalization layer for federated learning, addressing a real limitation of BN under non-IID data and small batch sizes. The paper has clear strengths: it releases code, provides a self-contained derivation of the unbiased global statistics identity in Appendix A.2, and runs extensive experiments across datasets, heterogeneity levels, batch sizes, client numbers, architectures, and FL optimizers. The central idea of separating the update of statistical parameters from learnable parameters is well motivated. However, the reported experiments operate in a regime where the exact unbiased-statistics guarantee does not directly apply, and the evaluation protocol is underspecified in a way that could affect the fairness of the comparisons. These issues need to be addressed before the central claims can be accepted.

major comments (3)
  1. [Section 3.2 / Appendix A.2 / Section 4.1] The unbiased-statistics guarantee in Eq. (8) is derived under full participation: every client k ∈ [K] contributes its local statistics computed on its complete dataset D_k. The experiments, however, use C = 0.1 (10 of 100 clients per round) and replace Eq. (8) with the exponentially moving average in Eq. (11) using λ = 0.01. In the reported setup, the stored global statistics are therefore not the unbiased estimates of S(ω_g^{t-1}; D_g) that the paper's central contribution claims; they are a heavily smoothed aggregate over a small client subset. The paper should either (a) add experiments with full participation (C = 1, λ = 1) that directly exercise the unbiased mechanism, or (b) re-frame the headline claim and provide a bias analysis for partial participation. As written, the theoretical guarantee and the empirical regime do not match.
  2. [Section 4.1] The evaluation protocol is underspecified: the sentence "we test the statistical parameters of the same client subset for accuracy evaluation" does not state whether the test set covers all clients or only those clients selected in the last communication round. If the latter, HBN is evaluated on the same clients used to compute its final statistics, which can inflate its advantage over baselines whose statistics are evaluated on all clients. The authors must state explicitly which test distribution is used and, if only a subset is used, rerun the evaluation on the full test set or on held-out clients before the claim that HBN consistently outperforms existing normalization solutions can be accepted.
  3. [Section 4.3, Table 6] Ablation row (2), labeled "Equation (8) ← weighted average", is ambiguous because Eq. (8) is itself a weighted average with a between-client correction term. The description in the text, "summarise local statistics based on weighted average", does not identify what replacement is actually implemented. If the row replaces Eq. (8) with the naive weighted average of local statistics from Eq. (6), this should be stated explicitly; if not, the reader cannot tell which design choice is being ablated. This matters because the row is used to argue that the unbiased aggregation formula is a load-bearing component.
minor comments (5)
  1. [Section 4.3 / Table 6] In the ablation description, "moving average without separation" for row (1) is clear enough, but the distinction between rows (1) and (2) should be made precise in terms of which quantities are uploaded and how they are aggregated, so that the reader can reproduce the ablation configuration.
  2. [Appendix A.1, Eq. (14)] The chain of equalities in Eq. (14) uses |Σ a_i| = Σ |a_i|, which is not an equality in general; it should be an inequality (triangle inequality). The qualitative conclusion about bias remains, but the displayed derivation is technically incorrect.
  3. [Section 4.3 / Table 3] The notation for the Dirichlet coefficient is inconsistent: the text uses ϕ throughout most of the paper, but Table 3 and Appendix B.3 use β. Please unify the notation.
  4. [Table 1] The Tiny-ImageNet results for FBN (4.90 and 3.76) are far below all other methods, including GN and LN; this suggests the learning-rate range may not have covered a workable setting for FBN, and the paper does not discuss this. A brief note or an extended search would strengthen the comparison.
  5. [Section 4.2 / Tables 1-5] The paper reports only point estimates of test accuracy without error bars or repeated-run standard deviations. Given that the reported gains are often 1-3 percentage points, reporting variance across seeds would help assess whether the improvements are significant.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: HBN's unbiased-statistics claim is an algebraic consequence of the frozen-model protocol, not a reduction to its own inputs.

full rationale

The central theoretical claim is that the aggregation rule in Eq. (8) yields unbiased global statistics S(omega_g^{t-1}; D_g). The paper proves this in Appendix A.2: when each client computes its local statistics on its full local dataset with the previous global model frozen (Eqs. 7 and 15), the weighted aggregation is exactly the mean and variance of that global model over the union of all client data (Eq. 16). This is an identity derived from the protocol's definitions, not a fitted parameter relabeled as a prediction, and it depends on no self-citation. The learnable hybrid factor alpha is trained on the training objective and ablated in Table 6, so it is not assumed into existence. The only overlapping self-citation (Zhong et al., 2023, whose author list includes H.-Y. Chen) is used to describe the FixBN baseline and is not load-bearing for HBN's contributions. The paper does contain a genuine empirical limitation: the unbiasedness identity requires full participation and full local data, while the experiments activate C=0.1 clients and replace Eq. (8) with the moving-average update in Eq. (11); Section 4.1 also states that evaluation uses the statistics of the same client subset. These are validity and reproducibility concerns about the experimental protocol, not circular reasoning in the derivation chain. Accordingly, no circular step is exhibited, and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on a standard statistical identity (pooled mean/variance), a full-participation assumption that is violated in the experiments, and the paper's hypothesis that historical global statistics help normalize current training batches. The method adds two tunable quantities: the learned mixing factor alpha and the hand-set moving-average momentum lambda. No new physical or conceptual entities are postulated.

free parameters (2)
  • hybrid distribution factor alpha (per channel, per client)
    Learned via SGD on each client's local data; controls the sigmoid-weighted mixing of batch and global statistics in Eq (9). The method's benefit depends on this learned balance.
  • moving-average momentum lambda for global statistics = 0.01
    Set by hand in Section 4.1; used in Eq (11) to smooth global statistics under partial participation. With 10% client participation, this biased estimator replaces the unbiased Eq (8), so the chosen value affects the validity of the 'unbiased' claim.
assumptions (3)
  • domain assumption All clients participate in every round when computing global statistics
    Appendix A.2 derives Eq (8) as unbiased with full participation; experiments use C=0.1, so only 10% of clients contribute statistics, violating this premise.
  • ad hoc to paper The global statistics of the previous round's model remain useful for normalizing the current round's training batches
    The hybrid normalization in Eq (9) mixes batch statistics with the stored global statistics from omega^{t-1}_g; the benefit of this design is asserted and tested empirically, not derived.
  • domain assumption Client data partitions are fixed and each client's data distribution is stable across rounds
    The pooled-variance formula requires the client datasets and their sizes N_k to be the reference population; in practice, partial participation and stragglers make the aggregated statistics an approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning." pith.science (2026). https://pith.science/paper/7OYLWAR7

@misc{pith2026250521877,
  author       = {Pith},
  title        = {Pith review of: Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OYLWAR7}},
  note         = {Machine review of arXiv:2505.21877}
}
read the original abstract

Batch Normalisation (BN) is widely used in conventional deep neural network training to harmonise the input-output distributions for each batch of data. However, federated learning, a distributed learning paradigm, faces the challenge of dealing with non-independent and identically distributed data among the client nodes. Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance. To this end, it is urgent to explore an alternative normalisation solution for federated learning. In this work, we resolve the dilemma of the BN layer in federated learning by developing a customised normalisation approach, Hybrid Batch Normalisation (HBN). HBN separates the update of statistical parameters (i.e. , means and variances used for evaluation) from that of learnable parameters (i.e. , parameters that require gradient updates), obtaining unbiased estimates of global statistical parameters in distributed scenarios. In contrast with the existing solutions, we emphasise the supportive power of global statistics for federated learning. The HBN layer introduces a learnable hybrid distribution factor, allowing each computing node to adaptively mix the statistical parameters of the current batch with the global statistics. Our HBN can serve as a powerful plugin to advance federated learning performance. It reflects promising merits across a wide range of federated learning settings, especially for small batch sizes and heterogeneous data.

Figures

Figures reproduced from arXiv: 2505.21877 by the authors.

Figure 1
Figure 1. A comparison of our Hybrid Batch Normalisation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Normalisation methods for two toy FL clusters. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Different Dirichlet coefficients ϕ(0.6, 0.1) to label distribution on CIFAR-10 with 100 clients. (GN), Layer Normalisation (LN), FedFN, FixBN, and Fed￾erated Batch Normalisation (FBN), For fairness, we default to combining all normalisation methods with the baseline FedAvg. Benchmark. All experiments are conducted on the classic image classification datasets, including CIFAR-10/CIFAR￾100 (Krizhevsky, 2009) and Tiny-… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The sensitivity of different normalisation methods to batch size on CIFAR-100 with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Impact of the hyper-parameter λ under different client activation ratios C. CIFAR-10 with ϕ = 0.6, using Simple-CNN with a batch size of 32. We keep other settings consistent with those in Section 4.1. When the client activation rate is low, adding a moving average can…
Figure 6
Figure 6. Figure 6: Experiments on CIFAR-100 (β = 0.6, B = 32) across varying local epochs by Simple-CNN. B.3. Different local training epochs [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Convergence curves of HBN under different model architectures on CIFAR-100 with [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 4 canonical work pages

  1. [1]

    Each client k ∈ [K] downloads the global model as the initialisation of the local model, and trains it on local dataset Dk. Due to the fact that the statistical parameters in vanilla BN are updated synchronously with the learnable parameters, the statistical parameters obtained in each locality are approximate estimates of ωt k = ωt−1 g − η∇L(ωt−1 g , Dk)...

  2. [5]

    H., Qi, H., and Brown, M

    Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual clas- sification. arXiv preprint arXiv:1909.06335,

  3. [6]

    Batch normalization: Accelerating deep net- work training by reducing internal covariate shift

    Ioffe, S. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167,

  4. [9]

    On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,

    Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data.arXiv preprint arXiv:1907.02189,

  5. [10]

    Fedbn: Federated learning on non-iid features via local batch normalization

    Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021b. Lubana, E. S., Dick, R., and Tanaka, H. Beyond batchnorm: Towards a unified understanding of normalization in deep learning. NeurIPS, 34:4778–4791,

  6. [12]

    Continual normalization: Rethinking batch normalization for online continual learn- ing

    Pham, Q., Liu, C., and Hoi, S. Continual normalization: Rethinking batch normalization for online continual learn- ing. arXiv preprint arXiv:2203.16102,

  7. [15]

    Xu, J., Wang, S., Wang, L., and Yao, A. C.-C. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874,

  8. [16]

    Federated learning with non-iid data.arXiv preprint arXiv:1806.00582,

    Zhao, Y ., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V . Federated learning with non-iid data.arXiv preprint arXiv:1806.00582,

Show all 18 references
  1. [2009]

    One weird trick for parallelizing convolu- tional neural networks

    Krizhevsky, A. One weird trick for parallelizing convolu- tional neural networks. arXiv preprint arXiv:1404.5997,

  2. [2015]

    BN can effectively solve this problem under centralised training

    refers to the change in the input distribution of each layer during the training of deep neural networks due to updates in model parameters. BN can effectively solve this problem under centralised training. However, in the context of federated learning, the changes in model pa...

  3. [2017]

    arXiv preprint arXiv:1706.02677,

  4. [2018]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,

  5. [2019]

    Overcoming the challenges of batch normalization in federated learning

    Guerraoui, R., Pinot, R., Rizk, G., Stephan, J., and Taiani, F. Overcoming the challenges of batch normalization in federated learning. arXiv preprint arXiv:2405.14670,

  6. [2020]

    Ba, J. L. Layer normalization. arXiv preprint arXiv:1607.06450,

  7. [2021]

    Towards un- derstanding regularization in batch normalization

    Luo, P., Wang, X., Shao, W., and Peng, Z. Towards un- derstanding regularization in batch normalization. arXiv preprint arXiv:1809.00846,

  8. [2022]

    Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Koneˇcn`y, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arXiv preprint arXiv:2003.00295,

  9. [2023]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

  10. [2024]

    A downsam- pled variant of imagenet as an alternative to the cifar datasets

    Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsam- pled variant of imagenet as an alternative to the cifar datasets. ArXiv, abs/1707.08819,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.