{"id":"2cc211ba-c7e0-4852-94f8-c0ec4d6ae31f","arxiv_id":"2411.12244","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Wrapping Optuna and Ray Tune into a PySyft federated learning pipeline with per-round client feedback yields higher test accuracy than random search on FEMNIST and CIFAR10, while the proposed step-wise feedback mechanism itself is not ablated.","lead":"The authors integrate two off-the-shelf hyperparameter optimization tools, Optuna and Ray Tune, into a PySyft-based federated learning pipeline and report higher test accuracy than a random search baseline on FEMNIST and CIFAR10. They also propose a step-wise feedback mechanism and a client grouping strategy to speed up tuning, though neither is tested in isolation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates the proposed step-wise feedback mechanism: all experiments compare Optuna/RayTune (with step-wise) against random search, so the reported gains cannot be attributed to the claimed mechanism. This leaves the central claim unsupported.","rationale":"The reader's REJECT is appropriate. My focus differs slightly from the reader's stated weakest assumption: rather than asking whether marginal feedback can misrank under HP interactions, I ask whether the experiments provide any evidence that the mechanism is present and effective. This is the most load-bearing concern because the central claim is specifically about the step-wise mechanism, and the reported experiments cannot distinguish it from the HPO toolkits' default search behavior. Section 5 claims the mechanism is effective, but Tables 4-5 compare only tool identity, not mechanism on/off. Algorithm 1 also omits the marginal evaluations described in Section 3.2, so the implementation is under-specified. A clean ablation would settle the claim: if disabling step-wise feedback leaves accuracy unchanged, the mechanism is superfluous; if enabling it with Random Search does not help, the mechanism is not a general accelerator. The absence of code and the use of test-set feedback in the HPO objective are additional independent problems, but they are secondary to the missing isolation of the claimed contribution. Because the central claim is unsupported by the presented evidence, the rejection stands.","tokens_in":15140,"tokens_out":6758,"duration_ms":70573,"concrete_test":"Run the same FEMNIST/CIFAR10 pipeline with two additional conditions: (i) Optuna and RayTune using only global validation loss as feedback (step-wise mechanism disabled), and (ii) Random Search with the step-wise mechanism enabled. Compare final test accuracy and total wall-clock time across all four conditions with matched search budgets. If (i) matches the reported Optuna/RayTune accuracies, or if (ii) does not beat plain Random Search, the step-wise mechanism contributes nothing and the central claim is unsupported. Report at least three seeds per condition, since no error bars are given.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 is that 'this step-wise mechanism, when integrated with HPO toolkits, is effective for finding more optimum configurations.' But the experiments in Tables 4-5 and Figure 4 vary only the search algorithm (Random Search vs. Optuna vs. RayTune); the step-wise feedback mechanism is held constant in the two toolkit conditions and absent (or unspecified) in the Random Search baseline. There is no condition that runs Optuna or RayTune without step-wise feedback, and no condition that adds step-wise feedback to Random Search. Without those controls, the differences in accuracy and runtime could be entirely due to the off-the-shelf samplers in Optuna/RayTune. The problem is compounded by the incomplete specification of the mechanism: Section 3.2 says each client sends feedback for the current HP combination plus 'the next best choice of each HP,' but Algorithm 1 line 8 trains only the current h_j and line 16 passes only aggregate losses to the HPO tool; there is no description of how 'next-best' values are generated, how those marginal evaluations are performed, or how the HPO tool consumes them. Thus the evidence does not even establish that the mechanism was implemented as described, let alone that it is responsible for the reported improvements. The claimed 10%/20% speedup from the client-selection scheme is likewise presented without a runtime table or ablation. A rejection is warranted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes integrating off-the-shelf hyperparameter optimization (HPO) toolkits, Optuna and RayTune, into a federated learning (FL) pipeline built on PySyft with FedAvg. It introduces a 'step-wise adaptive mechanism' in which clients send per-round local and global loss feedback to the HPO tool during training, together with a dynamic client-grouping strategy intended to mitigate stragglers and speed up search. Experiments on FEMNIST and CIFAR10 compare Optuna and RayTune against a random-search baseline, reporting higher test accuracy and faster search for the toolkits.","tokens_in":15410,"tokens_out":5251,"duration_ms":58822,"significance":"If the step-wise mechanism were properly evaluated and shown to improve upon the same HPO toolkits without it, the paper would offer a practical, lightweight recipe for Auto-FL on resource-constrained devices. The paper is useful as a system-design discussion: it surveys AutoML toolkits, identifies FL-specific constraints, and proposes a low-fidelity search strategy. However, the central algorithmic claim is not supported by the current experimental design: all comparisons vary both the HPO toolkit and the step-wise mechanism simultaneously, the reported test accuracies are the quantity being optimized, and the mechanism itself is underspecified in the pseudocode. The contribution is therefore best treated as a proposal requiring substantially stronger evidence, rather than as an established result.","major_comments":[{"comment":"The central claim, stated in Section 5 as 'this step-wise mechanism, when integrated with HPO toolkits, is effective for finding more optimum configurations,' is not tested by the experiments. Every comparison is Optuna/RayTune (with step-wise feedback) versus Random Search (without it), so the reported accuracy and speed improvements are confounded by the choice of HPO toolkit and by the step-wise mechanism. There is no condition running Optuna or RayTune without the step-wise feedback, and no condition adding step-wise feedback to Random Search. Without these controls, the results cannot attribute the observed gains to the proposed mechanism.","section":"Section 5, Tables 4–5, Figure 4"},{"comment":"The described mechanism is not implemented in the pseudocode. Section 3.2 says each client sends feedback for the current hyperparameter combination plus 'the next best choice of each HP, holding the other constant,' but Algorithm 1, line 8, trains only with the current h_j and line 12 aggregates a single loss `l([ch_j])`; nowhere does the algorithm evaluate or transmit marginal per-hyperparameter feedback. The text also does not define how 'next best' values are generated, how many additional local training steps are used, or how the HPO tool consumes these marginal losses. As written, the pseudocode does not match the described method, so the reader cannot verify that the reported experiments actually used the step-wise mechanism.","section":"Section 3.2, Algorithm 1"},{"comment":"The search objective is circular with the reported results. Section 4.4 states that 'the mean validation loss and average test accuracy have been selected as the objective function for HPO,' and Section 3.2 says clients send both training and testing losses while the global server also sends global training and test loss. The final reported accuracies in Tables 4 and 5 are therefore the same quantities optimized during the search, not independent test results. The HPO search should use validation feedback only, with a held-out test set used solely for final evaluation; otherwise the reported accuracies can be optimistically biased.","section":"Section 4.4, Section 3.2"},{"comment":"The claimed speedups—'up to 10% and 20% faster compared to Random Search for Optuna and RayTune, respectively'—are presented without any runtime table, variance estimates, or statistical test. The client-selection/grouping strategy from Section 3.3 is also never evaluated in isolation, so its purported contribution to the speedup is unsupported. The authors should report wall-clock times, number of communication rounds, and results across multiple random seeds, and ideally include ablations that isolate the client-selection component.","section":"Section 5, 'client selection' discussion"}],"minor_comments":[{"comment":"The experimental scale is described inconsistently: Section 4.2 says CIFAR10 is partitioned among 50 clients, Table 3 lists `client_num` as 200 for FEMNIST and 5 for CIFAR10, and Figure 4's caption refers to large-scale settings with 200 clients. The exact number of clients and data partition should be made consistent throughout.","section":"Section 4.2, Table 3, Figure 4"},{"comment":"There are several presentation issues: Table 4 uses 'Raytuna' for RayTune; Table 3's title says 'Auto-ML toolkits' though the table lists FL libraries; Table 6 lists 'Drop out [0.1−0−5]' which likely should be `[0.1, 0.5]`; references [1] and [2] are the same paper, as are [17] and [18]; and the text refers to 'Figures 4 and 5' but only Figure 4 appears.","section":"Tables and references"},{"comment":"The averaging formula in line 12 is ambiguous: it writes `lf_j <- 1/c_j sum_{c=1}^{c_j}(l([ch_j]) + lf_j)`, mixing `lf_j` on both sides and using an undefined `c_j` and bracketed `[ch_j]`. This needs a clean mathematical definition or a reference to a clearly stated equation.","section":"Algorithm 1, line 12"}],"recommendation":"major_revision","confidential_remarks":"I see no reason to doubt the authors' good faith, but the experimental protocol needs fundamental rework: controlled ablations (same toolkit with and without step-wise feedback), test-set separation, multiple seeds, and full runtime statistics. The duplicate references and inconsistent table titles suggest the manuscript was prepared hastily. If the authors can provide the missing ablations and clarify the algorithm, the paper could become a useful systems contribution; as it stands, the central claim is not demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper is a genuine engineering effort: Optuna and RayTune are wired into a PySyft/FedAvg pipeline, and both beat a random-search baseline on FEMNIST and CIFAR10 in the authors' setup. That is a useful demonstration for practitioners. But the main advertised novelty, the step-wise adaptive feedback mechanism, is never tested in isolation. The Optuna/RayTune conditions always include step-wise feedback; the random-search baseline does not. There is no condition that runs the tools without the step-wise feedback and no condition that adds step-wise feedback to random search. The accuracy gains could therefore be due entirely to the off-the-shelf samplers.\n\nWhat is genuinely new here is narrow: the specific integration of two HPO libraries into a PySyft FL loop, with per-round client feedback and a client-grouping scheme to accommodate stragglers. The survey of AutoML toolkits and the discussion of what makes an HPO tool suitable for FL is competent and may help someone choose a library. That's real value.\n\nThe soft spots are serious rather than cosmetic. First, the objective function is stated as mean validation loss and average test accuracy, and the feedback described in Section 3.2 includes training and testing loss. That means the reported final test accuracies are partly the quantity the search is optimizing, so the headline numbers are fitted, not independent. Second, Algorithm 1 does not match the prose: it trains only the current hyperparameter set and passes aggregate losses; the 'next best choice of each HP' evaluations described in Section 3.2 are absent. Without code or a complete algorithm, readers cannot tell whether the step-wise mechanism was actually implemented as described. Third, all results are single runs, and the claimed 10%/20% speedup has no runtime table behind it.\n\nThere are also presentation issues: duplicated references, malformed table entries, typos. Those are minor.\n\nWho gets value from this? Someone who wants a starting template for Optuna/RayTune inside PySyft. Not someone who needs evidence for feedback-driven HPO in FL. As submitted, the evidence does not support the central claim. I would not send this to peer review; it needs a proper ablation, error bars, and a rerun with the objective and feedback restricted to validation data before it is refutable. If those are provided, it could become a modest empirical note.","headline":"Useful integration demo, but the claimed step-wise mechanism is never isolated, test accuracy is fed back into the search, and Algorithm 1 doesn't match the described mechanism—so the central claim is unsupported.","tokens_in":15949,"tokens_out":4619,"would_cite":false,"duration_ms":48609,"reading_group":"no","serious_thinker":"unclear","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A step-wise local feedback mechanism lets off-the-shelf HPO toolkits outperform random search when tuning hyperparameters in federated learning.","keywords":["federated learning","hyperparameter optimization","AutoML","Optuna","Ray Tune","step-wise feedback","client selection","non-IID data"],"falsifier":"Run the same step-wise feedback pipeline on a search space with a known strong interaction between two hyperparameters (for example, learning rate and weight decay, where the optimum lies on a diagonal ridge), and compare the configurations it selects against an exhaustive grid over that same space. If the step-wise mechanism's chosen configurations consistently have higher validation loss than the median grid configuration, the ranking assumption is falsified.","tokens_in":14922,"feed_emoji":"⚙️","tokens_out":5728,"duration_ms":52961,"temperature":0.7,"pith_summary":"The paper claims that standard hyperparameter-optimization (HPO) toolkits such as Optuna and RayTune can be made effective in federated learning if the server feeds them a step-wise signal: each client reports how the recent local loss changes when one hyperparameter is moved to its next best value while the others stay fixed. That signal, averaged with the global validation loss, narrows the search space and lets the toolkit choose better configurations than random search in the same number of communication rounds. On non-IID FEMNIST and CIFAR10, the mechanism improves test accuracy over a random-search baseline for both toolkits, with Optuna reaching about 81% accuracy on FEMNIST and about 73% on CIFAR10. If right, the approach makes automated tuning practical on resource-constrained edge clients because it avoids evaluating many full hyperparameter combinations.","feed_headline":"Step-wise feedback finds better FL hyperparameters than random search","feed_subtitle":"Optuna and RayTune hit higher test accuracy on non-IID FEMNIST and CIFAR10 with less tuning time.","key_machinery":"The step-wise adaptive feedback mechanism is the central object. For each hyperparameter being tuned, each client computes the local training loss for the current configuration and for the configuration that moves that one hyperparameter to its next-best value; this per-hyperparameter marginal feedback is what allows the HPO toolkit to narrow the search space after a single round instead of waiting for full evaluations. The server's weighted averaging of local feedbacks with the global validation loss, weighted by the number of participating nodes, is what coordinates the generic toolkit with the federated learning loop. Dynamic client grouping based on training time also feeds multiple parallel evaluations into the toolkit, mitigating the straggler effect.","core_discovery":"The central claim is that a step-wise adaptive feedback mechanism, integrated with generic HPO toolkits on a trusted federated server, finds more optimal hyperparameter configurations than any other combination in the same search space does under random search. In each communication round, every client trains locally for a short step and returns not only the loss of the current hyperparameter set but also the loss obtained by perturbing each hyperparameter to its next-best candidate value, holding the other hyperparameters fixed. The server averages these local feedbacks with the global validation loss, weighting the global signal by the number of participating nodes, and passes the result to the HPO toolkit, which then proposes the next configuration. Because the toolkit can update its suggestion after every round rather than after a full evaluation, the search is accelerated; the paper reports that Optuna and RayTune complete training up to 10% and 20% faster than random search respectively, and that both produce higher test accuracy on FEMNIST and CIFAR10.","pith_inferences":["The step-wise mechanism behaves like a coordinate-descent search with noisy gradients; testing wider perturbation steps or evaluating multiple candidate values per hyperparameter could reveal how sensitive the ranking is to interaction effects.","The same feedback signal could be used to detect which hyperparameters matter most for a given client cohort, potentially enabling per-cluster or personalized hyperparameter schedules without additional communication.","If the marginal-ranking assumption holds, the method should transfer to other federated optimizers like FedProx with minimal change, since it only modifies the feedback channel, not the aggregation rule.","A natural benchmark extension would compare against Bayesian-optimization-based HPO under a fixed communication budget rather than only random search, to see whether the step-wise signal adds value on top of a smarter sampler."],"forward_implications":["Off-the-shelf HPO tools like Optuna and RayTune can be dropped into an FL pipeline without redesigning the federated algorithm, as long as the server is trusted and can collect the extra local feedback.","Hyperparameter tuning in FL can move from offline grid or random search to per-round online updates, cutting the number of communication rounds needed to reach a given accuracy.","The client-selection strategy reduces the straggler effect, so the speedup holds when clients have heterogeneous compute and network resources.","The approach targets local hyperparameters such as learning rate, epochs, and weight decay; extending it to global and communication-management hyperparameters would require HPO tools that model FL-specific parameters.","Because only lightweight feedback is exchanged, the method is compatible with resource-constrained edge devices and privacy constraints that forbid sharing raw data."],"supporting_citations":[{"why":"Supplies the finding that using only the most recent performance feedback is sufficient, which motivates the step-wise rather than history-aggregating update.","marker":"[13]"},{"why":"Introduces weight-sharing for FL hyperparameter optimization (FedEx), the baseline approach the step-wise mechanism is contrasted with and built upon.","marker":"[12]"},{"why":"Industrial FL evaluation of grid search and Bayesian HPO that defines the local-versus-global HPO comparison this paper extends.","marker":"[10]"},{"why":"FLoRA, a single-shot FL-HPO method for non-neural models, provides a reference point for how HPO is adapted to federated constraints.","marker":"[26]"},{"why":"Random search baseline that requires 1200 communication rounds per HP evaluation, which this paper's per-round feedback mechanism is designed to beat.","marker":"[17]"},{"why":"Survey of FL challenges on resource-constrained IoT devices, motivating the need for lightweight HPO and straggler-aware client selection.","marker":"[11]"}],"fun_headline_variants":["Step-wise feedback accelerates FL hyperparameter search by 20%","Adaptive HPO finds better federated learning settings than random search","Optuna and RayTune with step-wise feedback beat random search in FL","Faster and more accurate hyperparameter tuning for federated learning","Step-wise adaptive mechanism speeds up Auto-FL tuning on edge devices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a short local-training loss measured for a one-step change in a single hyperparameter, averaged together with the global validation loss, ranks hyperparameter configurations in the same order as the true federated objective.","fun_headline_variants_meta":{"raw":{"variants":["Step-wise feedback accelerates FL hyperparameter search by 20%","Adaptive HPO finds better federated learning settings than random search","Optuna and RayTune with step-wise feedback beat random search in FL","Faster and more accurate hyperparameter tuning for federated learning","Step-wise adaptive mechanism speeds up Auto-FL tuning on edge devices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1608,"prompt_tokens":1034,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":650,"tokens_out":574,"duration_ms":6228,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:45:53.423690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same step-wise feedback pipeline on a search space with a known strong interaction between two hyperparameters (for example, learning rate and weight decay, where the optimum lies on a diagonal ridge), and compare the configurations it selects against an exhaustive grid over that same space. If the step-wise mechanism's chosen configurations consistently have higher validation loss than the median grid configuration, the ranking assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the finding that using only the most recent performance feedback is sufficient, which motivates the step-wise rather than history-aggregating update."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces weight-sharing for FL hyperparameter optimization (FedEx), the baseline approach the step-wise mechanism is contrasted with and built upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Industrial FL evaluation of grid search and Bayesian HPO that defines the local-versus-global HPO comparison this paper extends."},{"cited_title":"FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning","cited_arxiv_id":"2112.08524","evidence_quote":"FLoRA, a single-shot FL-HPO method for non-neural models, provides a reference point for how HPO is adapted to federated constraints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Survey of FL challenges on resource-constrained IoT devices, motivating the need for lightweight HPO and straggler-aware client selection."}],"review_version":1}