{"id":"f1b1e4d1-3e92-4292-baa7-83687318c858","arxiv_id":"1908.07420","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Federated learning aggregation can be improved by weighting clients with label diversity and model divergence, but tuning the priority order on test accuracy overstates the benefit.","lead":"This paper proposes a device-aware federated learning aggregation that weights each participant by multiple criteria instead of dataset size alone. It also tunes the criteria priority order online using test accuracy, a choice that weakens the validity of the reported gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The online priority adjustment selects models using the same local test sets later used to compute the reported round counts, so the 12-round gain over FedAvg is not an unbiased estimate.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the online adjustment uses the local test accuracy both for selecting the priority ordering and for the reported outcome, creating selection bias. In Algorithm 1, the local test sets are used at every accepted or rejected round, and the final metric is the number of rounds to reach target accuracy on those same test sets. This concern is central because the largest claimed improvement comes from Study C, not from the static multi-criteria results. A held-out validation split for the online search and an untouched evaluation split for reporting would separate tuning from assessment and would also allow error bars. The paper's static MCA results (e.g., Ds≻Ld≻Md at 20 rounds vs 22 for FedAvg at 75% for 20% of devices) are still worth reporting as preliminary evidence, but they are weaker than the claimed substantial improvement. The paper also lacks a formal convergence proof, but the main block to accepting the headline is the evaluation design, not the mathematics of the aggregation operator. With the requested re-evaluation, the paper could be acceptable as a methods proposal; the current experiment alone does not support the strong empirical claim.","tokens_in":10917,"tokens_out":6564,"duration_ms":65847,"concrete_test":"Split each client's local test set into a validation split and an evaluation split, with no sample overlap. Run Algorithm 1 unchanged except that lines 13-27 use validation-split accuracy for the online priority search, and report the round count on the evaluation split only once training completes. Repeat on multiple 10% subsamples and report the mean and standard error for Final and FedAvg. If the Final row's advantage over FedAvg shrinks or reverses, the Study C result is an artifact of test-set peeking.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline empirical claim is the Study C result that the Final variant with online adjustment reaches 75% accuracy for 20% of devices in 12 rounds versus 22 for FedAvg. Algorithm 1, lines 13-16, estimates global accuracy by averaging local test accuracies, and lines 17-27 search over priority orderings to maximize that same accuracy. Table 1 then reports the number of rounds needed to reach target accuracy on those exact local test sets. Selection and evaluation therefore share the same data, and with six permutations per round over up to 1,000 rounds even a small amount of test-set-based selection can systematically lower the measured round count. The paper also reports the best of six initializations for Final without error bars or repeated subsamples, so the 12-vs-22 comparison has no variance estimate. This does not impugn the static multi-criteria idea, where the gains are more modest, but it means the paper's strongest claimed advantage is not established by the reported experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a device-aware federated learning framework. Instead of weighting local model updates only by local dataset size as in FedAvg, it computes each client's contribution from a set of normalized criteria (dataset size, label diversity, and local model divergence) using a prioritized multi-criteria aggregation operator. It further introduces an online adjustment procedure (Algorithm 1) that, at each communication round, searches over priority orderings of the criteria and selects the ordering that maximizes a weighted average of local test accuracies. Experiments on a 10% subsample of the FEMNIST dataset compare FedAvg, each criterion in isolation, all six static priority permutations, and the online-adjusted variant, reporting the number of rounds needed to reach 75% or 80% target accuracy for specified percentages of devices.","tokens_in":11087,"tokens_out":7122,"duration_ms":71207,"significance":"If the empirical claims were robust, the paper would be a useful practical contribution: it demonstrates how domain knowledge about clients can be incorporated into FL aggregation through a standard prioritized aggregation operator, and the idea of self-adjusting the priority order online is appealing. The static multi-criteria study is transparent in reporting all six permutations, and the use of the non-IID FEMNIST benchmark is appropriate. However, the headline advantage of the online-adjustment variant is currently not established, because the selection criterion and the reported evaluation metric are both computed on the same local test sets, and all results come from single runs without variance estimates. With a corrected evaluation protocol and repeated trials, the proposed framework could still be valuable.","major_comments":[{"comment":"The online-adjustment evaluation is circular. Algorithm 1 selects the priority ordering at each round by maximizing a weighted average of local test accuracies (lines 13-16), searching over permutations until the estimated accuracy no longer improves (lines 17-27). Table 1 then reports the number of rounds needed to reach target accuracy on those same local test sets. The reported advantage of the Final variant over FedAvg (e.g., 12 vs 22 rounds at 20% of devices for 75% accuracy) therefore reflects selection on the test data rather than an unbiased estimate of generalization. The paper should select parameters on a held-out validation split and evaluate on a separate test split, or at minimum report the current numbers as an optimistic upper bound. Notably, the Final mean over the six initializations is 20.5 rounds versus the static MCA mean of 22.5, which is a much smaller gap than the best-run 12-round result.","section":"Section 4, Study C; Algorithm 1, lines 13-27; Table 1"},{"comment":"All results in Table 1 are based on single runs, and the Final variant additionally reports the best of six initializations. Without repeated seeds, multiple subsamples, or any variance estimate, differences such as 25.5 vs 27 rounds or 405 vs 552.5 rounds cannot be distinguished from noise, and reporting the best initialization as the headline is optimistic. The paper should provide means and standard deviations (or full distributions) over several independent runs and should justify any use of the best run rather than the mean.","section":"Section 4, Table 1; Section 3, Experimental setup"}],"minor_comments":[{"comment":"In the second part of Example 1, the numerical computation is inconsistent with Equation (4): for the order C3, C2, C1, the weights are λ1=1, λ2=0.9, λ3=0.9·0.8=0.72, so sk = 1·0.9 + 0.9·0.8 + 0.72·0.5 = 1.98, not 1.82 as printed.","section":"Section 2.2, Example 1"},{"comment":"Footnote 2 states '0.2×317=75 devices'; this should presumably be based on the 371 participating clients, so it should read '0.2×371≈74', and '317' appears to be a typo.","section":"Footnote 2"},{"comment":"The phrase 'desired target global accuracy of 75% and 80%' is confusing because the table columns report the percentage of devices reaching a target accuracy, not a single global accuracy value; please rephrase to avoid ambiguity.","section":"Section 4, first paragraph"},{"comment":"The prose after Algorithm 1 says that when no further permutations improve accuracy, 'the candidate global model which produced the least worst test accuracy is assigned to wG_{t+1}', but Algorithm 1 lines 22-24 select the ordering with the maximum accuracy. These two descriptions should be reconciled.","section":"Section 2.2 vs Algorithm 1"},{"comment":"The statement 'we use a subsampled version (10% of total, 371 clients involved)' is ambiguous because 10% of the 3,550 FEMNIST writers would be 355, not 371; please specify exactly how the subsample was drawn.","section":"Section 3, Federated dataset"},{"comment":"In the abstract, 'pushing forward the state the art in the field' should be 'pushing forward the state of the art in the field.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core prioritized-aggregation idea is plausible, and the static multi-criteria results may be salvageable, but the current experimental evaluation does not support the paper's strongest claim about online adjustment. The authors need to re-run the Study C experiments with a proper held-out validation split and provide variance estimates across repeated runs. This is a substantial but fixable evaluation issue, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is importing the prioritized multi-criteria aggregation operator from da Costa Pereira et al.'s IR work into federated learning client weighting, plus the online search over priority orderings. That combination is new as far as I know. The static multi-criteria results (Study MCA) are the more credible part: they show modest gains over FedAvg from using label diversity and model divergence as additional criteria, with dataset size still leading in the low/mid participation regime. That is a reasonable empirical claim, and the paper is honest that FEMNIST is a 10% subsample for computational reasons.\n\nThe soft spot is exactly what the stress-test note says. The online adjustment (Algorithm 1, lines 16-27) selects the priority ordering that maximizes estimated global accuracy on local test sets, and then Table 1 reports the number of rounds needed to reach target accuracy on those same local test sets. The headline Final result — 12 rounds vs 22 for FedAvg at 75% for 20% of devices — is therefore not an unbiased estimate of the method's benefit. With six initializations and no error bars, the best-of-six reporting makes it worse. I don't think this invalidates the static idea, but it means the paper's strongest claimed advantage is not established by the experiment as designed. Separating tuning from evaluation (e.g., a validation round or holdout test set) and reporting variance across subsamples would fix most of this.\n\nAlso worth flagging, though minor: the paper says Ld's 405 mean beats base Ds's 552.5 by 36%, but that is actually about 27%. And the text says 0.2×317 while the dataset has 371 clients. Small errors, but they are in the numbers the reader is supposed to trust.\n\nThe citation pattern looks fine: FedAvg, LEAF, aggregation operators, and convergence-related work are all standard. No code is shipped, which limits reproducibility but is not unusual for a paper of this vintage.\n\nBottom line: this deserves a serious referee. The core idea is defensible and the static results are worth reporting, but the online adjustment needs a cleaner evaluation before the headline claim can be believed. A revision that separates tuning from testing and adds variance estimates would make this a decent workshop-to-conference paper.\n\nIf you are working on weighted aggregation in FL, this is worth a quick look for the prioritized operator trick. I would not cite the Final result, but I might cite the static multi-criteria comparison as related work.","headline":"A plausible device-aware FL weighting idea, undermined by evaluation design: the online priority search selects on the same test sets used to report the headline round counts.","tokens_in":11602,"tokens_out":1270,"would_cite":false,"duration_ms":15800,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that federated models train faster when each client's update is weighted by a prioritized, adaptively reordered combination of dataset size, label diversity, and model divergence, rather than by dataset size alone.","keywords":["federated learning","aggregation operator","prioritized aggregation","device-aware weighting","non-IID data","model divergence","label diversity","online adjustment"],"falsifier":"Rerun the fully adjusted scheme on the same public benchmark, holding out a disjoint evaluation slice of each client's data that the online adjustment never sees, and count rounds to target accuracy on that slice; if the 12-versus-22 advantage over standard federated averaging disappears, the gain comes from selecting on the local test sets rather than from the criteria themselves.","tokens_in":10704,"feed_emoji":"📱","tokens_out":9744,"duration_ms":92682,"temperature":0.7,"pith_summary":"Federated learning lets many devices train a shared model without uploading raw data, and the standard averaging rule weights each device's update only by how much local data it holds. This paper argues that throwing away other measurable properties—how many different labels a device covers and how far its local model drifts from the global model—costs accuracy and communication efficiency. It proposes scoring devices with a prioritized multi-criteria aggregation operator, then letting the server adjust the criteria's priority order round by round using a local search that keeps whichever ordering yields the best estimated accuracy. In experiments on a public federated handwriting benchmark, the fully adjusted scheme reaches 75% accuracy for 20% of devices in 12 communication rounds, against 22 for standard federated averaging, and it improves the average rounds-to-target at every device fraction tested. The result matters because communication rounds are the scarce resource in federated training; better aggregation weights can reduce them without touching the privacy guarantee.","feed_headline":"Device-aware weighting hits 75% accuracy in 12 rounds vs 22","feed_subtitle":"Prioritized multi-criteria aggregation reaches target accuracy in fewer communication rounds.","key_machinery":"The load-bearing mechanism is the prioritized multi-criteria aggregation operator, which converts each client's criterion evaluations $c^k_1,\\ldots,c^k_m$ into a device score $$s_k=\\sum_{i=1}^m \\lambda_i\\, c^k_{(i)}, \\qquad \\lambda_1=1,\\quad \\lambda_i=\\lambda_{i-1}\\,c^k_{(i-1)},$$ where $c^k_{(i)}$ is the evaluation of the $i$-th criterion after sorting criteria by a priority order from most to least important. This recursive weight rule makes a poorly satisfied high-priority criterion dampen the influence of every lower-priority criterion, so a client cannot compensate for failing an important property by excelling at a minor one. The companion mechanism is the online adjustment loop of Algorithm 1: the server computes a candidate global model under the current priority order, tests it on local test sets, and if estimated accuracy has not improved, backtracks through other permutations until one improves accuracy or the least-bad candidate is kept. Together these two mechanisms let the server reweight clients every round without seeing their raw data.","core_discovery":"The paper's central claim is that the weight each client receives in federated averaging should encode more than local dataset size, and that a prioritized combination of three criteria—dataset size, label diversity, and local model divergence—produces a better global model than the standard size-only weighting. It further claims that the priority order among these criteria should not be fixed once by a domain expert; the server can adjust it online, choosing at each round the ordering whose candidate global model gives the highest estimated accuracy on local test sets, and backtracking when accuracy drops. On the public federated handwriting benchmark used in the experiments, the fully adjusted device-aware scheme reaches 75% accuracy for 20% of devices in 12 communication rounds, versus 22 rounds for standard federated averaging, and at high device fractions it reaches the same target in 329 mean rounds against 552.5 for the size-only baseline. The authors frame the result as evidence that aggregation weights are a meaningful design axis for federated learning.","pith_inferences":["The same prioritized reweighting principle should transfer to other federated applications, such as recommendation or language modeling, as long as domain-specific criteria can be measured without exposing raw user data.","With more than a handful of criteria the exhaustive permutation search would grow factorially, so practical adoption would likely need a cheaper heuristic or a learned policy for choosing priority orders.","The priority order chosen online could double as an interpretable diagnostic, revealing which device properties dominate at different stages of training—a use the paper leaves implicit."],"forward_implications":["If device-aware weighting is adopted, federated training can reach a target accuracy in fewer communication rounds: the best configuration needs 12 rounds to 75% accuracy for 20% of devices, versus 22 for standard federated averaging.","Criteria beyond dataset size—label diversity and model divergence—are worth measuring, since in isolation they already outperform the size-only baseline when a high fraction of devices must reach the target.","The best priority order depends on the goal: dataset size first is best for quickly serving a small fraction of devices, while model divergence first is best for a large fraction, so no single hand-picked ordering dominates.","Online adjustment with backtracking improves over all static permutations on average, making the scheme less sensitive to the initial priority ordering chosen by the domain expert."],"supporting_citations":[{"why":"Introduces the FedAvg algorithm and the size-only weighting rule that the paper extends, and supplies the experimental hyperparameter configuration.","marker":"[13]"},{"why":"Defines the prioritized multi-criteria aggregation operator with the recursive importance weights used to score each device.","marker":"[6]"},{"why":"Provides the federated benchmark platform and the realistically distributed handwriting dataset used in all experiments.","marker":"[3]"},{"why":"Motivates the model-divergence criterion by analyzing how local updates can drift away from the global objective.","marker":"[15]"},{"why":"Motivates criteria beyond dataset size by showing how non-IID local data degrades federated model performance.","marker":"[18]"}],"fun_headline_variants":["Adaptive multi-criteria weighting hits 75% accuracy in 12 rounds vs 22","Dynamic client priorities cut FL convergence from 22 to 12 rounds","Priority-aware aggregation boosts FL to 75% accuracy in 12 rounds","Self-adjusting aggregation weights reach 75% accuracy in 12 rounds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The online adjustment step assumes that choosing the priority ordering that maximizes accuracy on the clients' local test sets also improves the true global model; if the search instead overfits to those particular test examples, the reported gains are an artifact of peeking.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive multi-criteria weighting hits 75% accuracy in 12 rounds vs 22","Dynamic client priorities cut FL convergence from 22 to 12 rounds","Priority-aware aggregation boosts FL to 75% accuracy in 12 rounds","Self-adjusting aggregation weights reach 75% accuracy in 12 rounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4035,"prompt_tokens":940,"completion_tokens":3095,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":3012}},"tokens_in":556,"tokens_out":3095,"duration_ms":21700,"temperature":1.0,"reasoning_tokens":3012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:18:41.765430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the fully adjusted scheme on the same public benchmark, holding out a disjoint evaluation slice of each client's data that the online adjustment never sees, and count rounds to target accuracy on that slice; if the 12-versus-22 advantage over standard federated averaging disappears, the gain comes from selecting on the local test sets rather than from the criteria themselves.","supporting_citations":[{"cited_title":"In: Proceedings of the 20th International Conference on Artific ial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdal e, FL, USA","cited_arxiv_id":null,"evidence_quote":"Introduces the FedAvg algorithm and the size-only weighting rule that the paper extends, and supplies the experimental hyperparameter configuration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the prioritized multi-criteria aggregation operator with the recursive importance weights used to score each device."}],"review_version":1}