Pith. sign in

REVIEW 5 major objections 6 minor 26 references

Federated Learning with Sample-level Client Drift Mitigation

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that client drift in federated learning is the accumulated effect of per-sample biases, and that training clients on low-loss samples first and progressively adding high-loss samples reduces drift and improves accuracy…

desk verdict A plausible sample-level curriculum for FL with broad empirical gains, but the drift-mitigation mechanism itself is never actually tested. read the letter →

arxiv 2501.11360 v1 pith:6FLOK7Y2 submitted 2025-01-20 cs.LG cs.AI

Haoran Xu , Jiaze Li , Wanyi Wu , Hao Ren This is my paper · ORCID
classification cs.LGcs.AI
keywords federatedlearningclientdriftsample-levelbiasprogressivelabeldistributionskewfeaturenoisylabelsuncertainty-basedselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that client drift in federated learning is not a single monolithic error but the cumulative effect of per-sample biases, with each local sample contributing differently to how the local model departs from the global model. The authors propose FedBSS, a two-stage method: a warm-up stage that trains on all samples to build diverse knowledge, then a progressive stage that ranks a client's samples by their loss under the global model, splits them into 'unbiased' and 'biased' sets using a maximum-uncertainty boundary, and adds biased samples back epoch by epoch on a cosine schedule. The claim is that this sample-level ordering reduces drift and improves global accuracy under label skew, feature skew, and noisy labels, without extra communication between clients and server.

What carries the argument

The central mechanism is FedBSS's two-stage schedule. Stage 1 is a diversified knowledge acquisition warm-up of fixed rounds (50 in the experiments) where the client trains on all samples. Stage 2, in each round, evaluates every local sample with the global model, sorts by loss, and sets the split point between unbiased and biased samples at the sample with maximum uncertainty $\alpha(x;\vartheta)=1-(\max p - \min p)$. Training then starts from the unbiased set and adds biased samples progressively through $\alpha = (1-\cos(e/e_{\mathrm{total}}\pi))/2$ per local epoch. The cosine schedule matters because the loss distribution is dense near the threshold and sparse at the edges, so a linear ramp fits the sample curve poorly. This mechanism is what converts the per-sample bias ordering into an actual training schedule.

What would settle it

On a heterogeneous benchmark, compute each sample's actual contribution to client drift (for example, the norm of the per-sample gradient projected against the global-update direction) and compare that ordering to the loss ordering FedBSS uses; if the two orderings disagree substantially on high-loss samples, FedBSS will be seen to add the wrong samples, and its accuracy should drop below FedAvg on a dataset engineered so that high-loss samples are the informative rare class.

Watch

Extended reading notes

Core claim

The central claim is that client drift can be understood and mitigated at the level of individual samples. The paper identifies that each local sample carries a different degree of bias, that the loss of a sample under the current global model measures that bias, and that the bias ordering changes as training progresses. FedBSS operationalizes this by sorting client samples by global-model loss, using the sample with maximum uncertainty as the adaptive threshold between low-bias ('unbiased') and high-bias ('biased') samples, and training first on low-loss samples while progressively mixing in high-loss samples with a cosine ramp. The paper reports that this outperforms existing drift-calibration and aggregation methods across Fashion-MNIST, CIFAR-10, CIFAR-100, and DomainNet, and remains effective when labels are noisy.

Load-bearing premise

The load-bearing premise is that a sample's loss under the current global model faithfully measures how much that sample pushes the local update away from the global update; if loss ordering and drift contribution diverge, the schedule keeps harmful samples and discards useful ones.

Editorial extensions

If this is right

  • If the central claim holds, client drift can be reduced by ordering samples within each client, making the method orthogonal to update-calibration and aggregation techniques, so it can be stacked with them.
  • The method requires no extra communication; the only added cost is locally sorting samples by loss, so it is applicable to communication-constrained federated systems.
  • The improvement grows with heterogeneity and with model and dataset scale, suggesting the benefit is largest precisely in the settings where client drift is most severe.
  • Under label noise the method's robustness increases with the noise ratio, implying that the biased-sample grouping absorbs noisy samples and contains their drift contribution.
  • The adaptive uncertainty threshold removes the need for a fixed loss threshold across rounds, clients, and datasets, so the method does not require tuning a per-task threshold.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the fixed 50-round warm-up with an uncertainty-stability criterion; the paper's own ablation shows performance is sensitive to warm-up length, but the paper does not derive a principled stopping rule.
  • The 'train low-loss first' ordering is a federated analogue of curriculum learning; it could be tested as a drop-in local schedule in centralized training, where the drift notion is replaced by generalization disparity.
  • Because high-loss samples under the global model are also the samples the model is least certain about, the same loss-plus-uncertainty machinery could be reused for federated active learning, selecting which samples to label or request from clients.
  • If loss ordering is noisy early in training, the uncertainty threshold may flicker across rounds; smoothing the threshold over time is a concrete testable tweak the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces FedBSS, a two-stage federated learning algorithm. In Stage 1 (warm-up), standard FL runs for T1 rounds. In Stage 2, each client computes per-sample losses under the current global model, ranks samples, splits them at the sample with maximum uncertainty (Eq. 5), and trains on the low-loss 'unbiased' set while progressively adding high-loss 'biased' samples according to the cosine schedule in Eq. (6). The authors argue that client drift is the cumulative effect of per-sample biases and that progressively learning from low-bias to high-bias samples mitigates drift. They report accuracy comparisons on Fashion-MNIST, CIFAR-10, and CIFAR-100 under Shards and Dirichlet label-skew partitions, on noisy-label versions of those datasets, and on DomainNet with feature skew.

Significance. If the mechanism were established, FedBSS would be a useful addition: it is communication-free beyond standard FedAvg, simple to implement, and the tables show consistent gains over several strong baselines, especially under high label skew and high noise. The authors also provide a schedule ablation (Fig. 5b) and a warm-up sensitivity study (Fig. 5a). However, the core conceptual claim—that loss under the global model measures per-sample drift contribution and that the maximum-uncertainty split (Eq. 5) separates unbiased from biased samples—is not directly tested. The empirical gains could arise from a generic curriculum effect rather than from drift mitigation. The DomainNet experiment is under-reported. These gaps limit the paper's significance until the mechanism is isolated.

major comments (5)
  1. [Motivation (Q1), Figure 3(a)] The experiment supporting the claim that 'loss is a measure of the sample's influence on client drift' trains separate models on the bottom-50%, top-50%, and 100% loss subsets. These conditions differ in both subset size and label composition, so the visualized drift differences do not isolate per-sample bias; the same pattern could appear with any random hard/easy split. Please add a size- and class-balanced control, and quantify drift (e.g., local-update deviation from the global update) rather than showing unlabeled trajectories.
  2. [Methodology, Eqs. (4)-(5) and Figure 3(b)] Figure 3(b) shows that uncertainty is non-monotone in loss: the highest-loss samples are often confidently wrong. Thus the maximum-uncertainty point does not delineate 'samples that contribute most to drift' from those that contribute little; it merely picks the most uncertain sample. The paper provides no evidence that this boundary is the correct unbiased/biased split. Please test the threshold rule against alternatives (e.g., fixed quantile, a loss-based boundary, or a boundary chosen by directly measured drift contribution) and report results separately.
  3. [Experiments, Figure 5(b), Tables 1-2] The only selection ablation varies the addition schedule (filter/linear/cosine), never the ranking criterion. To support the drift-mitigation explanation, hold Eq. (6)'s schedule fixed and compare global-model-loss ranking against (a) random ranking and (b) local-model-loss ranking. Without this ablation, the large gains in Tables 1 and 2 could be produced by any easy-to-hard curriculum that progressively adds samples, with no need for the claimed drift mechanism.
  4. [Implementation Details and Ablation (Warmup Stage)] The warm-up length T1=50 is chosen after an ablation on the benchmark datasets (Fig. 5a), and the text concedes that '50 rounds is not always optimal'. This makes T1 a tuned hyperparameter, yet it is fixed across all datasets and partitions. Please provide a principled selection rule or a per-setting tuning protocol, and state how T1 was chosen in Tables 1-2; otherwise the comparison includes benchmark-specific parameter fitting.
  5. [Result with Feature Distribution Skew, Figure 4] The DomainNet experiment reports only FedAvg and FedBSS, with no standard deviations, no baseline methods, and no numerical final global accuracy. The claim that FedBSS is effective under feature distribution skew is therefore not supported. Please add results for at least the main baselines (FedProx, Scaffold, FedLC, InCo) with multiple runs, and specify the per-client data composition.
minor comments (6)
  1. [Methodology] The sentences 'The specific comparison results are in section .' and 'More discussion is demonstrated in section .' have empty cross-references; fill them with the appropriate experiment or ablation subsection.
  2. [Algorithm 1, line 12] The aggregation step writes ϑ_t = (1/N) Σ_n ϑ^n_t although only a random subset N_t of clients is selected; clarify whether the average is over the selected clients only.
  3. [Table 2] The heading 'Noisy Label Radio' should be 'Noisy Label Ratio'; also state how label noise is generated (e.g., symmetric vs. asymmetric flip rates).
  4. [Figure 3(a) and Figure 3(d)] The axes in Figure 3(a) are unlabeled and the caption does not define the plotted quantities; Figure 3(d) is captioned 'Variance of uncertainty' but the text refers to abrupt changes in adaptive classification points. Please reconcile the caption with the text.
  5. [References and text] Acar et al. 2021a and 2021b are the same arXiv item and should be distinguished or merged; there are also typographical issues such as 'our methodFedBSS' and 'V ollgraf' in the Fashion-MNIST reference.
  6. [Equation (6)] The expression X^n_{t,e} = X^n_no + α X^n_bias is ambiguous: clarify whether this denotes a union of sample sets, a fractional subset of the biased set, or a weighted sampling distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the performance claims are measured on held-out data, and the loss-based sample ranking is a modeling assumption rather than a fitted input renamed as a prediction.

full rationale

The paper's derivation chain is not circular in the sense of reducing a prediction to its own inputs. FedBSS defines a sample-selection schedule: Eq. (3) sorts client samples by global-model loss, Eq. (5) sets an adaptive split point using maximum uncertainty, and Eq. (6) progressively admits the high-loss 'biased' samples during local epochs. These equations define the algorithm, but they do not define the outcome. The headline claims are test-set accuracies in Tables 1 and 2, which are measured on held-out data after training, so they are not forced by construction. The motivating claim that loss measures a sample's contribution to client drift is an empirical hypothesis; it is tested, albeit imperfectly, in Fig. 3a and could be wrong, but being wrong is not the same as being circular. The warm-up length and cosine schedule are chosen by ablations on the same benchmarks, which is ordinary hyperparameter selection, not a fitted quantity reused as a prediction. The citations to uncertainty-based active learning (Fuchsgruber et al. 2024; Zhu et al. 2008) are independent external prior work and are not used to impose a uniqueness conclusion. No load-bearing self-citation chain or imported uniqueness theorem appears. The manuscript has incomplete cross-reference placeholders such as 'section .', which is a writing defect rather than a circular step. Overall, the central performance claim has independent empirical content, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method's central assumptions are empirical heuristics rather than derived results; the warm-up length is a tuned hyperparameter, and the threshold and schedule are chosen via ablation without theoretical grounding.

free parameters (2)
  • Warm-up communication rounds T1 = 50
    Set to 50 rounds based on the ablation in Figure 5a; the paper notes the optimal value is dataset-dependent, so this is a tuned hyperparameter tied to the reported results.
  • Progressive sample-addition schedule shape = cosine ramp (1 - cos(e/etotal * pi))/2
    Chosen by hand and validated by comparing with linear and filtering in Figure 5b; the schedule form is an ad hoc design choice rather than a derived result.
assumptions (4)
  • domain assumption Sample loss under the current global model measures the sample's contribution to client drift.
    Invoked in Motivation Q1 and Eq. 3; no proof is provided, only a qualitative plot (Figure 3a).
  • ad hoc to paper The sample with maximum model uncertainty marks the correct split between unbiased and biased samples.
    Methodology, Eq. 5; the uncertainty definition (Eq. 4) is introduced for this purpose and no external justification is given.
  • ad hoc to paper Progressively adding biased samples with a cosine schedule (Eq. 6) reduces drift without losing necessary knowledge.
    Methodology, Eq. 6; validated only by an ablation (Figure 5b), not by theory.
  • domain assumption A warm-up stage of standard FL training stabilizes the uncertainty-based threshold and provides diversified knowledge.
    Methodology, warmup paragraph; supported only by ablation (Figure 5a).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Learning with Sample-level Client Drift Mitigation." pith.science (2026). https://pith.science/paper/6FLOK7Y2

@misc{pith2026250111360,
  author       = {Pith},
  title        = {Pith review of: Federated Learning with Sample-level Client Drift Mitigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6FLOK7Y2}},
  note         = {Machine review of arXiv:2501.11360}
}
read the original abstract

Federated Learning (FL) suffers from severe performance degradation due to the data heterogeneity among clients. Existing works reveal that the fundamental reason is that data heterogeneity can cause client drift where the local model update deviates from the global one, and thus they usually tackle this problem from the perspective of calibrating the obtained local update. Despite effectiveness, existing methods substantially lack a deep understanding of how heterogeneous data samples contribute to the formation of client drift. In this paper, we bridge this gap by identifying that the drift can be viewed as a cumulative manifestation of biases present in all local samples and the bias between samples is different. Besides, the bias dynamically changes as the FL training progresses. Motivated by this, we propose FedBSS that first mitigates the heterogeneity issue in a sample-level manner, orthogonal to existing methods. Specifically, the core idea of our method is to adopt a bias-aware sample selection scheme that dynamically selects the samples from small biases to large epoch by epoch to train progressively the local model in each round. In order to ensure the stability of training, we set the diversified knowledge acquisition stage as the warm-up stage to avoid the local optimality caused by knowledge deviation in the early stage of the model. Evaluation results show that FedBSS outperforms state-of-the-art baselines. In addition, we also achieved effective results on feature distribution skew and noise label dataset setting, which proves that FedBSS can not only reduce heterogeneity, but also has scalability and robustness.

Figures

Figures reproduced from arXiv: 2501.11360 by the authors.

Figure 1
Figure 1. Illustration of FedBSS. Our approach includes local and global progressive learning, shown by green and pink arrows. In the client’s local learning, there are three steps: 1) Samples are rated by loss; higher i in Si signifies greater loss (redder), lower means lesser (bluer). 2) Loss-based sorting identifies an adaptive threshold for classifying samples into ’unbiased’ (low loss) or ’bias’ (higher loss) sets. 3) Tr… view at source ↗
Figure 2
Figure 2. Different samples on client have a different degree [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Different impacts of various local samples on model drift. (b) The relationship between loss and uncertainty [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Result on DomainNet dataset and always sensitive to datasets and hyperparameters. The performance of our method FedBSS is usually more promi￾nent when there is high heterogeneity setting. This may be because the biased samples drifts the model to a large extent when th…
Figure 5
Figure 5. Figure 5: Ablation Study on feature Non-IID setting with feature distribution skew. We utilize domainNet dataset which contains six domains: clipart, infograph, painting, quickdraw, real, and sketch. We selected ten categories for the classification task and ResNet-18 as backbon…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [4]

    Acar, D. A. E.; Zhao, Y.; Navarro, R. M.; Mattina, M.; Whatmough, P. N.; and Saligrama, V. 2021 b . Federated Learning Based on Dynamic Regularization. arXiv:2111.04263

    arXiv 2021

  4. [5]

    Chan, Y.-H.; Zhou, R.; Zhao, R.; Jiang, Z.; and Ngai, E. C. H. 2024. Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning. arXiv:2308.11464

    work page Pith review arXiv 2024

  5. [6]

    Fuchsgruber, D.; Wollschläger, T.; Charpentier, B.; Oroz, A.; and Günnemann, S. 2024. Uncertainty for Active Learning on Graphs. arXiv:2405.01462

    arXiv 2024

  6. [7]

    Gao, L.; Fu, H.; Li, L.; Chen, Y.; Xu, M.; and Xu, C.-Z. 2022. FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. In IEEE Conference on Computer Vision and Pattern Recognition

    work page 2022

  7. [8]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition . In 2016 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) , 770--778. IEEE

    work page 2016

  8. [9]

    Jhunjhunwala, D.; Wang, S.; and Joshi, G. 2023. FedExP: Speeding Up Federated Averaging via Extrapolation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5

    work page 2023

Show all 26 references
  1. [10]

    B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A

    Kairouz, P.; McMahan, H. B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A. N.; Bonawitz, K.; Charles, Z.; Cormode, G.; Cummings, R.; D'Oliveira, R. G. L.; Rouayheb, S. E.; Evans, D.; Gardner, J.; Garrett, Z.; Gasc \' o n, A.; Ghazi, B.; Gibbons, P. B.; Gruteser, M.; Harchaoui...

    2021

  2. [11]

    P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A

    Karimireddy, S. P.; Kale, S.; Mohri, M.; Reddi, S.; Stich, S.; and Suresh, A. T. 2020. SCAFFOLD : Stochastic Controlled Averaging for Federated Learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Rese...

    2020

  3. [12]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  4. [13]

    Lee, G.; Jeong, M.; Shin, Y.; Bae, S.; and Yun, S.-Y. 2022. Preservation of the Global Knowledge by Not-True Distillation in Federated Learning. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems,...

    2022

  5. [14]

    Li, Q.; He, B.; and Song, D. 2021. Model-Contrastive Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2021

  6. [15]

    K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V

    Li, T.; Sahu, A. K.; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V. 2020. Federated Optimization in Heterogeneous Networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020

    2020

  7. [16]

    Li, X.; Jiang, M.; Zhang, X.; Kamp, M.; and Dou, Q. 2021. Fed \ BN \ : Federated Learning on Non- \ IID \ Features via Local Batch Normalization. In International Conference on Learning Representations

    2021

  8. [17]

    Li, X.-C.; and Zhan, D.-C. 2021. FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data. KDD '21, 995–1005. ISBN 9781450383325

    2021

  9. [18]

    U.; and Jaggi, M

    Lin, T.; Kong, L.; Stich, S. U.; and Jaggi, M. 2020. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351--2363

    2020

  10. [19]

    Louizos, C.; Reisser, M.; and Korzhenkov, D. 2024. A Mutual Information Perspective on Federated Contrastive Learning. arXiv:2405.02081

    2024 arXiv

  11. [20]

    McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS

    2017

  12. [21]

    Peng, X.; Bai, Q.; Xia, X.; Huang, Z.; Saenko, K.; and Wang, B. 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, 1406--1415

    2019

  13. [22]

    Wang, H.; Xu, H.; Li, Y.; Xu, Y.; Li, R.; and Zhang, T. 2024. Fed CDA : Federated Learning with Cross-rounds Divergence-aware Aggregation. In The Twelfth International Conference on Learning Representations

    2024

  14. [23]

    Xiao, H.; Rasul, K.; and Vollgraf, R. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. CoRR, abs/1708.07747

    2017 arXiv

  15. [24]

    Yang, Z.; Zhang, Y.; Zheng, Y.; Tian, X.; Peng, H.; Liu, T.; and Han, B. 2023. FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  16. [25]

    Zhang, J.; Li, Z.; Li, B.; Xu, J.; Wu, S.; Ding, S.; and Wu, C. 2022. Federated Learning with Label Distribution Skew via Logits Calibration. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 26311-...

    2022

  17. [26]

    Zhu, J.; Wang, H.; Yao, T.; and Tsou, B. K. 2008. Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification. In Scott, D.; and Uszkoreit, H., eds., Proceedings of the 22nd International Conference on Computational Linguistic...

    2008

  18. [27]

    Zhu, Z.; Hong, J.; and Zhou, J. 2021. Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 12878--...

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.