REVIEW 3 major objections 4 minor 27 references
Empirical Analysis of Asynchronous Federated Learning on Heterogeneous Devices: Efficiency, Fairness, and Privacy Trade-offs
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Asynchronous federated learning converges up to 10x faster, but the privacy burden falls unevenly on high-end devices, which incur up to 5x higher privacy loss.
desk verdict Empirically interesting and credible on efficiency/fairness, but the privacy-disparity numbers don't cohere with the paper's own accounting, so treat the 5x claim with caution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the coupling of two standard components: staleness-aware asynchronous aggregation (FedAsync's decay factor $\alpha_k = \alpha/(1+\tau_k)$, which down-weights old updates) and per-client Moments Accountant privacy accounting ($\varepsilon$ computed from a fixed noise scale, sampling probability $q$, and the number of rounds the client actually contributes). The argument runs through this coupling: because faster devices send more updates with low staleness, they receive larger aggregation weights and accumulate proportionally larger $\varepsilon$; slower devices send fewer, staler updates, so they get less influence and smaller $\varepsilon$, yet suffer more noise-induced accuracy loss. The trade-off is summarized by varying $\alpha$, which trades convergence speed against participation skew and privacy disparity.
What would settle it
Recomputing each Table 3 epsilon from the paper's measured participation percentages using Eqs. (7)-(8) with $\sigma$ fixed would settle it: if a device's epsilon is not proportional to how often it actually updated, or if FedAvg's epsilon is not identical across devices, the claim that privacy loss tracks participation fails.
Extended reading notes
Core claim
The central claim, on the paper's own terms, is that under realistic device heterogeneity asynchronous aggregation converts the straggler problem into an inequality problem: the same staleness-aware weighting that makes FedAsync fast also makes it unfair and privacy-uneven. Empirically, at alpha=0.6 high-end devices account for about 80% of updates while low-end devices fall below 5%, and the cumulative privacy loss at sigma=0.5 reaches roughly 35 for the fastest device versus below 10 for the slowest. Because each device's update count drives its Moments Accountant budget, the privacy burden tracks participation exactly, and uniform LDP noise does not equalize outcomes: low-end devices, whose few updates are stale and down-weighted, show up to about 16% local accuracy degradation while high-end devices degrade by roughly 6%.
Load-bearing premise
The headline privacy-disparity numbers rest on the assumption that each device's reported epsilon was driven by that device's real update count, an assumption the paper's own FedAvg row (identical epsilon for all five devices despite documented low-end dropouts) leaves unverified.
Editorial extensions
If this is right
- If the paper is right, reported convergence speedups of asynchronous FL should be read together with the per-device participation skew; the 10x wall-clock gain is purchased by letting fast hardware dominate the update stream.
- Uniform local differential privacy noise does not deliver uniform privacy or uniform utility; low-end clients bear disproportionate accuracy loss under the same noise scale.
- The aggregation strength $\alpha$ acts as a fairness-privacy dial: lowering $\alpha$ narrows participation and privacy gaps at the cost of convergence speed.
- Per-client privacy accounting that uses real update counts would show the same disparity in any asynchronous FL deployment where participation is hardware-bound.
- Systems that need both speed and equity should weight updates by capacity or staleness while calibrating noise per client, rather than applying one-size-fits-all LDP.
Reading between the lines
- The paper does not state this, but its epsilon equations imply that the privacy disparity is a participation fact rather than a hardware fact: any asynchronous mechanism that changes who sends updates changes who pays the privacy cost.
- A direct extension of the authors' own suggestion would be to inverse-calibrate per-client noise, adding more noise to frequent high-end contributors and less to rare low-end ones, and test whether the epsilon spread narrows without pushing global accuracy below the FedAvg level.
- Because the paper reports participation, staleness, and epsilon together, one could define a composite privacy-fairness index equal to epsilon divided by influence; under the reported numbers this index would be near-constant if the accountant tracks participation, which would make the disparity purely a weighting artifact.
- Comparing with client-level differential privacy instead of LDP would isolate whether low-end accuracy loss comes from per-message noise or from stale-model training; the paper does not run this control.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports an empirical comparison of synchronous FedAvg and staleness-aware FedAsync on a physical testbed of five edge devices (Raspberry Pi 3/4, NXP HummingBoard) for speech emotion recognition on CREMA-D. Local updates are perturbed with Gaussian noise under a DP-SGD-style mechanism, and per-client privacy loss is tracked with a custom Moments Accountant. The paper claims that FedAsync reaches 75% accuracy roughly 9-10x faster in wall-clock time than FedAvg, that high-end devices contribute 6-10x more updates and incur up to 5x higher per-client epsilon, and that low-end devices suffer greater accuracy degradation. The efficiency and participation measurements are plausible and largely internally consistent, but the privacy-loss accounting is not reconciled with the measured participation and dropout behavior, so the quantitative privacy-disparity claims are not supported as reported.
Significance. If the privacy accounting is corrected, the central message—that asynchronous FL improves wall-clock convergence while shifting both influence and privacy burden toward high-end devices—would be a useful empirical contribution for systems designers choosing aggregation and privacy mechanisms. The paper's strengths are the physical testbed spanning five hardware tiers, multi-seed averaging, direct wall-clock convergence measurements, and explicit per-device participation, staleness, and accuracy breakdowns. The fairness and efficiency findings are more solid than the privacy findings. However, the paper does not release code, per-client update-count logs, or the custom accountant implementation, and the privacy-loss numbers are the part of the paper that most needs independent verification.
major comments (3)
- [Section 4.2.1, Table 2; Section 4.2.3, Table 3; Section 3.2, Eqs. (7)-(8)] The FedAvg row in Table 3 reports a single epsilon for 'All Devices' at each sigma, yet Section 4.2.1 and Table 2 document that HW T1 dropped out three times and HW T2 dropped out twice during FedAvg training. Under the per-client accountant of Eqs. (7)-(8), a client that misses rounds must have strictly fewer composed moments and hence strictly lower cumulative epsilon than a client that never drops out. Equal epsilon for all devices is therefore only consistent with an accountant fed a nominal fixed round count rather than measured per-client participation. Since the same accounting pipeline is used for FedAsync, the per-client epsilon values in Table 3 are not shown to follow from the update counts measured in Section 4.2.2. This directly undermines the headline claim of up to 5x higher privacy loss for high-end devices; the authors should release per-client update-count logs and the accountant code, and recompute Table 3 from the measured participation, or explicitly reconcile the FedAvg row with the documented dropouts.
- [Section 3.2, Eqs. (4)-(8); Algorithm 1, lines 8-16] The privacy accounting adds one moment mu_t per client round, but DP-SGD as written in Eqs. (4)-(6) applies clipping and Gaussian noise to every mini-batch. With E=1, B=128, and about 941 training samples per client, each round contains roughly eight noised mini-batch steps, and the Moments Accountant for DP-SGD must compose over those mini-batch steps rather than over whole rounds. If mu_t in Eq. (7) is intended to summarize an entire round, the paper must explain how the composition over mini-batches is folded into a single mechanism; as written, the reported epsilon values appear to undercount privacy loss by approximately the number of mini-batches per round. All absolute epsilon values in Table 3 are therefore questionable independently of the participation-mismatch issue.
- [Section 4.2.2, Fig. 5; Section 4.2.3, Table 3] The epsilon ratios in Table 3 do not track the participation ratios in Fig. 5 under the stated accountant. At alpha=0.2 and sigma=0.5, HW T5 has 31.87% participation versus 8.24% for HW T1 (ratio 3.87), while the reported epsilon ratio is 31.75/12.57 = 2.53; at alpha=0.6, the participation ratio is 42.34/4.50 = 9.41 but the epsilon ratio is 35.12/6.79 = 5.17. Sublinear RDP composition might explain the divergence, but the paper does not provide the per-update RDP parameters or the accountant trace needed to verify this. Because the headline privacy ratio is derived from these numbers, the authors should show the step-by-step accountant output for at least one alpha and one sigma value.
minor comments (4)
- [Section 6 and Abstract] The conclusion states that high-end clients experience 'up to 6x greater privacy loss', while the abstract and Section 4.2.4 state 'up to 5x'; Table 3 gives a maximum ratio of about 5.17, so the multiplier should be harmonized across the paper.
- [Section 4.1.4] The paper says FedAsync runs 'continuous updates' until convergence, but the total number of asynchronous updates and the per-client update counts are never stated; Table 3's epsilon values cannot be reproduced without these counts. A table or appendix listing update counts for each device and each alpha would resolve this.
- [Figure 4] The inset in Figure 4 uses sparse tick labels such as 67, 467, 867, 1267 without a clear indication of the scale; the inset axes should be labeled explicitly and the tick format made readable.
- [Section 3.2, Eq. (7)] Equation (7) does not precisely define D and D' for the local mechanism; the authors should state explicitly that they are neighboring local datasets differing in one sample, and clarify whether privacy is accounted at the sample level or the client level.
Circularity Check
Privacy-loss disparity is a deterministic rescaling of the measured participation imbalance, so the headline 5x privacy-loss claim reduces by construction to the update-count result; the FedAvg control row also conflicts with documented dropouts.
-
self definitional
[Section 3.2, Eqs. (7)-(8); Algorithm 1 lines 14-17; Table 3; Section 4.2.1]
"The cumulative moment and resulting privacy budget are: µ(λ)=Σ_{t=1}^T µ_t(λ), ε=min_λ((µ(λ)−log δ)/λ). Algorithm 1: Update µ^{(k)}(λ)←µ^{(k)}(λ)+µ^{(k)}_t(λ). Table 3: FedAvg All Devices 26.55±0.47. Section 4.2.1: HW T1 ... three observed dropouts; HW T2 ... two dropouts."
With σ and q fixed (Section 4.1.4: σ∈{0.5,1,1.5,2}, q≈0.136), every per-round log moment µ_t is the same function for each device; the only client-specific variable in Eq. (8) is the number of rounds T_k over which a client's moments are accumulated. Table 3's per-client ε under FedAsync is therefore a deterministic, monotone transform of the participation percentages already measured in Fig. 5. The abstract's 'up to 5x higher privacy loss' is thus the '6–10x more updates' result restated through the accountant's composition formula, not an independent empirical finding. The FedAvg control row reinforces that the accountant was not actually driven by measured participation: Section 4.2.1 reports HW T1/T2 dropouts, yet Table 3 lists one identical ε for 'All Devices'.
full rationale
The paper's efficiency and fairness findings (10x faster convergence, participation skew at different α) are empirical, self-contained, and not circular. The per-client privacy-loss disparity, however, is computed by Eq. (8) as a sum of per-step moments over each client's updates. Since the noise scale and sampling probability are held constant across clients, the reported ε values are a monotone function of the very participation counts measured in Fig. 5; the '5x higher privacy loss' claim therefore carries no empirical content beyond the already-measured update-count imbalance. The FedAvg row heightens this concern: Table 3 reports a single ε for all devices even though Section 4.2.1 documents dropouts for HW T1 and HW T2, indicating the accountant was not transparently driven by per-client measured participation. No load-bearing self-citation or imported uniqueness theorem appears; references [3] and [16] are background only. Because the central privacy-disparity claim reduces by construction to the participation input while the efficiency/fairness claims are independent, the circularity is partial, not total.
Assumptions & free parameters
free parameters (2)
- Staleness decay factor alpha =
0.2, 0.4, 0.6
- LDP Gaussian noise scale sigma =
0.5, 1.0, 1.5, 2.0
assumptions (3)
- domain assumption Per-client privacy loss is computed by composing one DP-SGD step per local epoch with sampling probability q=B/|D_k| and the same sigma for every device, and this composition is valid in the asynchronous setting.
- domain assumption The fixed batch size B=128 with q=B/|D_k| approximately 0.136 accurately models the privacy sampling in the local data loaders.
- domain assumption The five CREMA-D partitions are IID and equal-sized, so performance divergence between devices is attributed to hardware and aggregation rather than data distribution.
Cite this review
Pith. "Pith review of Empirical Analysis of Asynchronous Federated Learning on Heterogeneous Devices: Efficiency, Fairness, and Privacy Trade-offs." pith.science (2026). https://pith.science/paper/7VODBULU
@misc{pith2026250507041,
author = {Pith},
title = {Pith review of: Empirical Analysis of Asynchronous Federated Learning on Heterogeneous Devices: Efficiency, Fairness, and Privacy Trade-offs},
year = {2026},
howpublished = {\url{https://pith.science/paper/7VODBULU}},
note = {Machine review of arXiv:2505.07041}
}
read the original abstract
Device heterogeneity poses major challenges in Federated Learning (FL), where resource-constrained clients slow down synchronous schemes that wait for all updates before aggregation. Asynchronous FL addresses this by incorporating updates as they arrive, substantially improving efficiency. While its efficiency gains are well recognized, its privacy costs remain largely unexplored, particularly for high-end devices that contribute updates more frequently, increasing their cumulative privacy exposure. This paper presents the first comprehensive analysis of the efficiency-fairness-privacy trade-off in synchronous vs. asynchronous FL under realistic device heterogeneity. We empirically compare FedAvg and staleness-aware FedAsync using a physical testbed of five edge devices spanning diverse hardware tiers, integrating Local Differential Privacy (LDP) and the Moments Accountant to quantify per-client privacy loss. Using Speech Emotion Recognition (SER) as a privacy-critical benchmark, we show that FedAsync achieves up to 10x faster convergence but exacerbates fairness and privacy disparities: high-end devices contribute 6-10x more updates and incur up to 5x higher privacy loss, while low-end devices suffer amplified accuracy degradation due to infrequent, stale, and noise-perturbed updates. These findings motivate the need for adaptive FL protocols that jointly optimize aggregation and privacy mechanisms based on client capacity and participation dynamics, moving beyond static, one-size-fits-all solutions.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Communication- efficient learning of deep networks from decentralized data
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–
-
[2]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur ´elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning.Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021
work page 2021
-
[3]
Samaneh Mohammadi, Ali Balador, Sima Sinaei, and Francesco Flammini. Balancing privacy and performance in federated learning: a systematic literature review on methods and metrics.Journal of Parallel and Distributed Computing, page 104918, 2024
work page 2024
-
[4]
Client selection for federated learning with heterogeneous resources in mobile edge
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. InICC 2019-2019 IEEE international conference on communications (ICC), pages 1–7. IEEE, 2019
work page 2019
-
[5]
Federated learning with buffered asynchronous aggregation
John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. InInternational Conference on Artificial Intelligence and Statistics, pages 3581–3607. PMLR, 2022
work page 2022
-
[6]
Tifl: A tier-based federated learning system
Zheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex, Ali Anwar, Nathalie Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, and Yue Cheng. Tifl: A tier-based federated learning system. InProceedings of the 29th international symposium on high-performance parallel and distributed computing, pages 125–136, 2020. 14
work page 2020
-
[7]
Asynchronous federated optimization.arXiv preprint arXiv:1903.03934, 2019
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization.arXiv preprint arXiv:1903.03934, 2019
arXiv 1903
-
[8]
Local differential privacy for deep learning.IEEE Internet of Things Journal, 7(7):5827– 5842, 2019
Pathum Chamikara Mahawaga Arachchige, Peter Bertok, Ibrahim Khalil, Dongxi Liu, Seyit Camtepe, and Mo- hammed Atiquzzaman. Local differential privacy for deep learning.IEEE Internet of Things Journal, 7(7):5827– 5842, 2019
work page 2019
Show all 27 references
-
[9]
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016
2016
-
[10]
Linear queries estimation with local differential privacy
Raef Bassily. Linear queries estimation with local differential privacy. InThe 22nd International Conference on Artificial Intelligence and Statistics, pages 721–729. PMLR, 2019
2019
-
[11]
A review on speech emotion recognition: a survey, recent advances, challenges, and the influence of noise.Neurocomputing, 568:127015, 2024
Swapna Mol George and P Muhamed Ilyas. A review on speech emotion recognition: a survey, recent advances, challenges, and the influence of noise.Neurocomputing, 568:127015, 2024
2024
-
[12]
Enhancing smart home design with ai models: A case study of living spaces implementation review.Energies, 16(6):2636, 2023
Amjad Almusaed, Ibrahim Yitmen, and Asaad Almssad. Enhancing smart home design with ai models: A case study of living spaces implementation review.Energies, 16(6):2636, 2023
2023
-
[13]
Automatic speech emotion recognition using machine learning.Social Media and Machine Learning [Working Title], 2019
Leila Kerkeni, Youssef Serrestou, Mohamed Mbarki, Kosai Raoof, Mohamed Ali Mahjoub, and Catherine Cleder. Automatic speech emotion recognition using machine learning.Social Media and Machine Learning [Working Title], 2019
2019
-
[14]
Swadha Gupta, Parteek Kumar, and Raj Kumar Tekchandani. Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models.Multimedia Tools and Applications, 82(8):11365–11394, 2023
2023
-
[15]
Privacy implications of voice and speech analysis–information disclosure by inference.Privacy and Identity Management
Jacob Leon Kr ¨oger, Otto Hans-Martin Lutz, and Philip Raschke. Privacy implications of voice and speech analysis–information disclosure by inference.Privacy and Identity Management. Data for Better Living: AI and Privacy: 14th IFIP WG 9.2, 9.6/11.7, 11.6/SIG 9.2. 2 Internatio...
2019
-
[16]
Balancing privacy and accuracy in federated learning for speech emotion recogni- tion
Samaneh Mohammadi, Mohammadreza Mohammadi, Sima Sinaei, Ali Balador, Ehsan Nowroozi, Francesco Flammini, and Mauro Conti. Balancing privacy and accuracy in federated learning for speech emotion recogni- tion. In2023 18th Conference on Computer Science and Intelligence Systems ...
2023
-
[17]
Differentially private federated learning: A client level perspec- tive
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspec- tive. InNeurIPS Workshop on Private Multi-Party Machine Learning, 2017
2017
-
[18]
Speech emotion recognition with deep convolutional neural networks.Biomedical Signal Processing and Control, 59:101894, 2020
Dias Issa, M Fatih Demirci, and Adnan Yazici. Speech emotion recognition with deep convolutional neural networks.Biomedical Signal Processing and Control, 59:101894, 2020
2020
-
[19]
Light-sernet: A lightweight fully convolutional neural network for speech emotion recognition.arXiv preprint arXiv:2110.03435, 2021
Arya Aftab, Shivangi Sharma, and Swati Srivastava. Light-sernet: A lightweight fully convolutional neural network for speech emotion recognition.arXiv preprint arXiv:2110.03435, 2021
2021 arXiv
-
[20]
Enhancing emotion recognition through federated learning: A multimodal approach with convolutional neural networks.Applied Sciences, 14(4), 2024
Nikola Simic, Haseeb Yousaf, and Emiliano Casalicchio. Enhancing emotion recognition through federated learning: A multimodal approach with convolutional neural networks.Applied Sciences, 14(4), 2024. 15
2024
-
[21]
Personalized federated learning with differential privacy.IEEE Internet of Things Journal, 7(10):9530–9539, 2020
Rui Hu, Yuanxiong Guo, Hongning Li, Qingqi Pei, and Yanmin Gong. Personalized federated learning with differential privacy.IEEE Internet of Things Journal, 7(10):9530–9539, 2020
2020
-
[22]
Flower: A friendly federated learning research framework.arXiv preprint arXiv:2007.14390, 2020
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusm ˜ao, et al. Flower: A friendly federated learning research framework.arXiv preprint arXiv:2007.14390, 2020
2007 arXiv
-
[23]
Opacus: User-friendly differential privacy library in pytorch.arXiv preprint arXiv:2109.12298, 2021
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch.arXiv preprint arXiv:2109.12298, 2021
2021 arXiv
-
[24]
Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014
2014
-
[25]
Projected federated averaging with heterogeneous differential privacy.Proceedings of the VLDB Endowment, 15(4):828–840, 2021
Junxu Liu, Jian Lou, Li Xiong, Jinfei Liu, and Xiaofeng Meng. Projected federated averaging with heterogeneous differential privacy.Proceedings of the VLDB Endowment, 15(4):828–840, 2021
2021
-
[26]
Online client selection for asynchronous federated learning with fairness consideration.IEEE Transactions on Wireless Communications, 22(4):2493–2506, 2022
Hongbin Zhu, Yong Zhou, Hua Qian, Yuanming Shi, Xu Chen, and Yang Yang. Online client selection for asynchronous federated learning with fairness consideration.IEEE Transactions on Wireless Communications, 22(4):2493–2506, 2022
2022
-
[27]
Adaptive personalized federated learning
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020. 16
2003 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.