REVIEW 4 major objections 3 minor 26 references
Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism
T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A step-wise local feedback mechanism lets off-the-shelf HPO toolkits outperform random search when tuning hyperparameters in federated learning.
desk verdict Useful integration demo, but the claimed step-wise mechanism is never isolated, test accuracy is fed back into the search, and Algorithm 1 doesn't match the described mechanism—so the central claim is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The step-wise adaptive feedback mechanism is the central object. For each hyperparameter being tuned, each client computes the local training loss for the current configuration and for the configuration that moves that one hyperparameter to its next-best value; this per-hyperparameter marginal feedback is what allows the HPO toolkit to narrow the search space after a single round instead of waiting for full evaluations. The server's weighted averaging of local feedbacks with the global validation loss, weighted by the number of participating nodes, is what coordinates the generic toolkit with the federated learning loop. Dynamic client grouping based on training time also feeds multiple parallel evaluations into the toolkit, mitigating the straggler effect.
What would settle it
Run the same step-wise feedback pipeline on a search space with a known strong interaction between two hyperparameters (for example, learning rate and weight decay, where the optimum lies on a diagonal ridge), and compare the configurations it selects against an exhaustive grid over that same space. If the step-wise mechanism's chosen configurations consistently have higher validation loss than the median grid configuration, the ranking assumption is falsified.
Extended reading notes
Core claim
The central claim is that a step-wise adaptive feedback mechanism, integrated with generic HPO toolkits on a trusted federated server, finds more optimal hyperparameter configurations than any other combination in the same search space does under random search. In each communication round, every client trains locally for a short step and returns not only the loss of the current hyperparameter set but also the loss obtained by perturbing each hyperparameter to its next-best candidate value, holding the other hyperparameters fixed. The server averages these local feedbacks with the global validation loss, weighting the global signal by the number of participating nodes, and passes the result to the HPO toolkit, which then proposes the next configuration. Because the toolkit can update its suggestion after every round rather than after a full evaluation, the search is accelerated; the paper reports that Optuna and RayTune complete training up to 10% and 20% faster than random search respectively, and that both produce higher test accuracy on FEMNIST and CIFAR10.
Load-bearing premise
The method assumes that a short local-training loss measured for a one-step change in a single hyperparameter, averaged together with the global validation loss, ranks hyperparameter configurations in the same order as the true federated objective.
Editorial extensions
If this is right
- Off-the-shelf HPO tools like Optuna and RayTune can be dropped into an FL pipeline without redesigning the federated algorithm, as long as the server is trusted and can collect the extra local feedback.
- Hyperparameter tuning in FL can move from offline grid or random search to per-round online updates, cutting the number of communication rounds needed to reach a given accuracy.
- The client-selection strategy reduces the straggler effect, so the speedup holds when clients have heterogeneous compute and network resources.
- The approach targets local hyperparameters such as learning rate, epochs, and weight decay; extending it to global and communication-management hyperparameters would require HPO tools that model FL-specific parameters.
- Because only lightweight feedback is exchanged, the method is compatible with resource-constrained edge devices and privacy constraints that forbid sharing raw data.
Reading between the lines
- The step-wise mechanism behaves like a coordinate-descent search with noisy gradients; testing wider perturbation steps or evaluating multiple candidate values per hyperparameter could reveal how sensitive the ranking is to interaction effects.
- The same feedback signal could be used to detect which hyperparameters matter most for a given client cohort, potentially enabling per-cluster or personalized hyperparameter schedules without additional communication.
- If the marginal-ranking assumption holds, the method should transfer to other federated optimizers like FedProx with minimal change, since it only modifies the feedback channel, not the aggregation rule.
- A natural benchmark extension would compare against Bayesian-optimization-based HPO under a fixed communication budget rather than only random search, to see whether the step-wise signal adds value on top of a smarter sampler.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes integrating off-the-shelf hyperparameter optimization (HPO) toolkits, Optuna and RayTune, into a federated learning (FL) pipeline built on PySyft with FedAvg. It introduces a 'step-wise adaptive mechanism' in which clients send per-round local and global loss feedback to the HPO tool during training, together with a dynamic client-grouping strategy intended to mitigate stragglers and speed up search. Experiments on FEMNIST and CIFAR10 compare Optuna and RayTune against a random-search baseline, reporting higher test accuracy and faster search for the toolkits.
Significance. If the step-wise mechanism were properly evaluated and shown to improve upon the same HPO toolkits without it, the paper would offer a practical, lightweight recipe for Auto-FL on resource-constrained devices. The paper is useful as a system-design discussion: it surveys AutoML toolkits, identifies FL-specific constraints, and proposes a low-fidelity search strategy. However, the central algorithmic claim is not supported by the current experimental design: all comparisons vary both the HPO toolkit and the step-wise mechanism simultaneously, the reported test accuracies are the quantity being optimized, and the mechanism itself is underspecified in the pseudocode. The contribution is therefore best treated as a proposal requiring substantially stronger evidence, rather than as an established result.
major comments (4)
- [Section 5, Tables 4–5, Figure 4] The central claim, stated in Section 5 as 'this step-wise mechanism, when integrated with HPO toolkits, is effective for finding more optimum configurations,' is not tested by the experiments. Every comparison is Optuna/RayTune (with step-wise feedback) versus Random Search (without it), so the reported accuracy and speed improvements are confounded by the choice of HPO toolkit and by the step-wise mechanism. There is no condition running Optuna or RayTune without the step-wise feedback, and no condition adding step-wise feedback to Random Search. Without these controls, the results cannot attribute the observed gains to the proposed mechanism.
- [Section 3.2, Algorithm 1] The described mechanism is not implemented in the pseudocode. Section 3.2 says each client sends feedback for the current hyperparameter combination plus 'the next best choice of each HP, holding the other constant,' but Algorithm 1, line 8, trains only with the current h_j and line 12 aggregates a single loss `l([ch_j])`; nowhere does the algorithm evaluate or transmit marginal per-hyperparameter feedback. The text also does not define how 'next best' values are generated, how many additional local training steps are used, or how the HPO tool consumes these marginal losses. As written, the pseudocode does not match the described method, so the reader cannot verify that the reported experiments actually used the step-wise mechanism.
- [Section 4.4, Section 3.2] The search objective is circular with the reported results. Section 4.4 states that 'the mean validation loss and average test accuracy have been selected as the objective function for HPO,' and Section 3.2 says clients send both training and testing losses while the global server also sends global training and test loss. The final reported accuracies in Tables 4 and 5 are therefore the same quantities optimized during the search, not independent test results. The HPO search should use validation feedback only, with a held-out test set used solely for final evaluation; otherwise the reported accuracies can be optimistically biased.
- [Section 5, 'client selection' discussion] The claimed speedups—'up to 10% and 20% faster compared to Random Search for Optuna and RayTune, respectively'—are presented without any runtime table, variance estimates, or statistical test. The client-selection/grouping strategy from Section 3.3 is also never evaluated in isolation, so its purported contribution to the speedup is unsupported. The authors should report wall-clock times, number of communication rounds, and results across multiple random seeds, and ideally include ablations that isolate the client-selection component.
minor comments (3)
- [Section 4.2, Table 3, Figure 4] The experimental scale is described inconsistently: Section 4.2 says CIFAR10 is partitioned among 50 clients, Table 3 lists `client_num` as 200 for FEMNIST and 5 for CIFAR10, and Figure 4's caption refers to large-scale settings with 200 clients. The exact number of clients and data partition should be made consistent throughout.
- [Tables and references] There are several presentation issues: Table 4 uses 'Raytuna' for RayTune; Table 3's title says 'Auto-ML toolkits' though the table lists FL libraries; Table 6 lists 'Drop out [0.1−0−5]' which likely should be `[0.1, 0.5]`; references [1] and [2] are the same paper, as are [17] and [18]; and the text refers to 'Figures 4 and 5' but only Figure 4 appears.
- [Algorithm 1, line 12] The averaging formula in line 12 is ambiguous: it writes `lf_j <- 1/c_j sum_{c=1}^{c_j}(l([ch_j]) + lf_j)`, mixing `lf_j` on both sides and using an undefined `c_j` and bracketed `[ch_j]`. This needs a clean mathematical definition or a reference to a clearly stated equation.
Circularity Check
The headline accuracy results are the same metric the HPO objective optimizes, so the central claim partly reduces to a fitted value rather than an independent test of the step-wise mechanism.
-
fitted input called prediction
[Section 4.4 'Implementation Details' and Section 5 'Experimental Result', Tables 4-5]
"The mean validation loss and average test accuracy have been selected as the objective function for HPO in this study. ... The results demonstrate that this step-wise mechanism, when integrated with HPO toolkits, is effective for finding more optimum configurations of HPs among all other HP combinations in the same search space."
The reported evidence for the step-wise mechanism is the test accuracy in Tables 4-5, but Section 4.4 states that average test accuracy is part of the HPO objective. The HPO search therefore selected hyperparameters to maximize exactly the quantity later reported as the demonstration of effectiveness. Saying the mechanism finds 'more optimum configurations' reduces to saying the optimizer attained higher values of its own objective. Moreover, the Random Search baseline lacks the step-wise mechanism while the Optuna/RayTune conditions differ in sampler as well, so the accuracy gain cannot be attributed to the mechanism. No independent hold-out metric or no-step-wise control is provided, making the central claim partially circular.
full rationale
The paper's main derivation is not a formal mathematical chain but an empirical evaluation of HPO tools. The clearest circularity is objective/reported-metric overlap: Section 4.4 defines the HPO objective as 'mean validation loss and average test accuracy,' and Section 5 uses the resulting test accuracy as the evidence that the step-wise mechanism is effective. Thus the reported accuracy is a fitted value from the search, not an independent prediction of the mechanism's merit. The comparison against Random Search is real, but it is confounded because the toolkit conditions always include the step-wise mechanism while Random Search does not, so the gain cannot be attributed specifically to the mechanism. Self-citations such as [3] and [11] are background references on FL and are not load-bearing for the central claim. No uniqueness theorem or self-citation chain forces the result. The central claim is therefore partially circular: the headline numbers reduce in part to the search objective itself, but the paper still provides a genuine empirical comparison, so the appropriate score is 6 rather than higher.
Assumptions & free parameters
free parameters (6)
- learning_rate =
0.01 (Optuna FEMNIST), 0.001 (Ray Tune FEMNIST)
- weight_decay =
1e-4 (Optuna FEMNIST), 1e-5 (Ray Tune FEMNIST), etc.
- local_epochs =
2 to 7 depending on method
- evaluation_frequency =
5 communication rounds
- global_feedback_weight =
not reported
- client_group_threshold =
not reported
assumptions (5)
- domain assumption FedAvg with local SGD is an appropriate optimizer for evaluating HPO tools
- domain assumption HPO tool surrogate models can consume averaged federated feedback without modification
- domain assumption The validation and test splits used for HPO feedback are disjoint from the final evaluation splits
- domain assumption The non-IID partitions approximate realistic federated heterogeneity
- domain assumption Random Search with PySyft defaults is a fair baseline
Cite this review
Pith. "Pith review of Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism." pith.science (2026). https://pith.science/paper/AB76HAM3
@misc{pith2026241112244,
author = {Pith},
title = {Pith review of: Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism},
year = {2026},
howpublished = {\url{https://pith.science/paper/AB76HAM3}},
note = {Machine review of arXiv:2411.12244}
}
read the original abstract
Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. While this privacy-preserving method is widely employed across various applications, it still requires significant development and optimization. Automated Machine Learning (Auto-ML) has been adapted for reducing the need for manual adjustments. Previous studies have explored the integration of AutoML with different FL algorithms to evaluate their effectiveness in enhancing FL settings. However, Automated FL (Auto-FL) faces additional challenges due to the involvement of a large cohort of clients and global training rounds between clients and the server, rendering the tuning process time-consuming and nearly impossible on resource-constrained edge devices (e.g., IoT devices). This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. A step-wise feedback mechanism has also been designed to accelerate the hyper-parameter tuning process and coordinate AutoML toolkits with the FL server. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process. Further, a novel client selection technique is introduced to mitigate the straggler effect in Auto-FL. The selected hyper-parameter tuning tools are evaluated using two benchmark datasets, FEMNIST, and CIFAR10. Further, the paper discusses the essential properties of successful HPO tools, the integration mechanism with the FL pipeline, and the challenges posed by the distributed and heterogeneous nature of FL environments.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[2]
Shaashwat Agrawal, Sagnik Sarkar, Mamoun Alazab, Praveen Kumar Reddy Maddikunta, Thippa Reddy Gadekallu, Quoc-Viet Pham, et al. 2021. Genetic CFL: hyperparameter optimization in clustered federated learning. Computational Intelligence and Neuroscience 2021 (2021)
work page 2021
-
[3]
Khandaker Mamun Ahmed, Ahmed Imteaj, and M Hadi Amini. 2021. Federated deep learning for heterogeneous edge computing. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) . IEEE, 1146–1152
work page 2021
-
[4]
Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. 2021. On large-cohort training for federated learning. Advances in neural information processing systems 34 (2021), 20461–20475
work page 2021
-
[5]
Luís Ferreira, André Pilastri, Carlos Manuel Martins, Pedro Miguel Pires, and Paulo Cortez. 2021. A comparison of AutoML tools for machine learning, deep learning and XGBoost. In 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 1–8
work page 2021
-
[6]
Marius Geitle and Roland Olsson. 2019. A new baseline for automated hyper-parameter optimization. In Machine Learning, Optimization, and Data Science: 5th International Conference, LOD 2019, Siena, Italy, September 10–13, 2019, Proceedings 5 . Springer, 521–530. , Vol. 1, No. 1, Article . Publication date: November 2024. 18 • Saadati and Amini
work page 2019
-
[7]
Nastaran Gholizadeh and Petr Musilek. 2022. Federated learning with hyperparameter-based clustering for electrical load forecasting. Internet of Things 17 (2022), 100470
work page 2022
-
[8]
Chaoyang He, Murali Annavaram, and Salman Avestimehr. 2020. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems 33 (2020), 14068–14080
2020
-
[9]
Chaoyang He, Erum Mushtaq, Jie Ding, and Salman Avestimehr. 2020. Fednas: Federated deep learning via neural architecture search. (2020)
work page 2020
Show all 26 references
-
[10]
Stephanie Holly, Thomas Hiessl, Safoura Rezapour Lakani, Daniel Schall, Clemens Heitzinger, and Jana Kemnitz. 2022. Evaluation of hyperparameter-optimization approaches in an industrial federated learning system. In Data Science–Analytics and Applications: Proceedings of the 4...
2022
-
[11]
Ahmed Imteaj, Urmish Thakker, Shiqiang Wang, Jian Li, and M Hadi Amini. 2021. A survey on federated learning for resource-constrained IoT devices. IEEE Internet of Things Journal 9, 1 (2021), 1–24
2021
-
[12]
Mikhail Khodak, Tian Li, Liam Li, Maria-Florina Balcan, Virginia Smith, and Ameet Talwalkar. 2020. Weight-Sharing for Hyperparameter Optimization in Federated Learning. In Int. Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML, Vol. 2020
2020
-
[13]
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina F Balcan, Virginia Smith, and Ameet Talwalkar. 2021. Federated hyperpa- rameter tuning: Challenges, baselines, and connections to weight-sharing. Advances in Neural Information Processing Systems 34 (2021), 19184–19197
2021
-
[14]
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning . PMLR, 6357–6368
2021
-
[15]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 (2020), 429–450
2020
-
[16]
Hesham Mostafa. 2019. Robust federated learning through representation matching and adaptive hyper-parameters. arXiv preprint arXiv:1912.13075 (2019)
2019 arXiv
-
[18]
Adrian Nilsson, Simon Smith, Gregor Ulm, Emil Gustavsson, and Mats Jirstrand. 2018. A performance evaluation of federated learning algorithms. In Proceedings of the second workshop on distributed infrastructures for deep learning . 1–8
2018
-
[19]
Sayan Putatunda and Kiran Rama. 2018. A comparative analysis of hyperopt as against other approaches for hyper-parameter optimization of XGBoost. In Proceedings of the 2018 international conference on signal processing and machine learning . 6–10
2018
-
[20]
Akshay Saini and Krishnan Ramanathan. 2021. FMS (Federated Model as a service) for healthcare: an automated secure-framework for personalized recommendation system. Cardiometry 20 (2021)
2021
-
[21]
Jonas Seng, Pooja Prasad, Devendra Singh Dhami, and Kristian Kersting. 2022. HANF: Hyperparameter And Neural Architecture Search in Federated Learning. arXiv preprint arXiv:2206.12342 (2022)
2022 arXiv
-
[22]
Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems 33 (2020), 21394–21405
2020
-
[23]
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems 33 (2020), 7611–7623
2020
-
[24]
Chunhui Zhang, Xiaoming Yuan, Qianyun Zhang, Guangxu Zhu, Lei Cheng, and Ning Zhang. 2021. Privacy-Preserving Neural Architecture Search Across Federated IoT Devices. In2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications (Trus...
2021
-
[25]
Huanle Zhang, Mi Zhang, Xin Liu, Prasant Mohapatra, and Michael DeLucia. 2021. Automatic Tuning of Federated Learning Hyper- Parameters from System Perspective. (2021)
2021
-
[26]
Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. 2021. Flora: Single-shot hyper-parameter optimization for federated learning. arXiv preprint arXiv:2112.08524 (2021)
2021 arXiv
-
[27]
Hangyu Zhu, Haoyu Zhang, and Yaochu Jin. 2021. From federated learning to federated neural architecture search: a survey. Complex & Intelligent Systems 7 (2021), 639–657
2021
-
[28]
Marc-André Zöller and Marco F Huber. 2021. Benchmark and survey of automated machine learning frameworks. Journal of artificial intelligence research 70 (2021), 409–472. , Vol. 1, No. 1, Article . Publication date: November 2024
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.