Pith. sign in

REVIEW 2 major objections 6 minor 18 references

Towards Effective Device-Aware Federated Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that federated models train faster when each client's update is weighted by a prioritized, adaptively reordered combination of dataset size, label diversity, and model divergence, rather than by dataset size alone.

desk verdict A plausible device-aware FL weighting idea, undermined by evaluation design: the online priority search selects on the same test sets used to report the headline round counts. read the letter →

arxiv 1908.07420 v1 pith:CVJCEMDX submitted 2019-08-20 cs.LG cs.DCstat.ML

Vito Walter Anelli , Yashar Deldjoo , Tommaso Di Noia , Antonio Ferrara This is my paper · ORCID
classification cs.LGcs.DCstat.ML
keywords federatedlearningaggregationoperatorprioritizeddevice-awareweightingnon-IIDdatamodeldivergencelabeldiversityonlineadjustment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated learning lets many devices train a shared model without uploading raw data, and the standard averaging rule weights each device's update only by how much local data it holds. This paper argues that throwing away other measurable properties—how many different labels a device covers and how far its local model drifts from the global model—costs accuracy and communication efficiency. It proposes scoring devices with a prioritized multi-criteria aggregation operator, then letting the server adjust the criteria's priority order round by round using a local search that keeps whichever ordering yields the best estimated accuracy. In experiments on a public federated handwriting benchmark, the fully adjusted scheme reaches 75% accuracy for 20% of devices in 12 communication rounds, against 22 for standard federated averaging, and it improves the average rounds-to-target at every device fraction tested. The result matters because communication rounds are the scarce resource in federated training; better aggregation weights can reduce them without touching the privacy guarantee.

What carries the argument

The load-bearing mechanism is the prioritized multi-criteria aggregation operator, which converts each client's criterion evaluations $c^k_1,\ldots,c^k_m$ into a device score $$s_k=\sum_{i=1}^m \lambda_i\, c^k_{(i)}, \qquad \lambda_1=1,\quad \lambda_i=\lambda_{i-1}\,c^k_{(i-1)},$$ where $c^k_{(i)}$ is the evaluation of the $i$-th criterion after sorting criteria by a priority order from most to least important. This recursive weight rule makes a poorly satisfied high-priority criterion dampen the influence of every lower-priority criterion, so a client cannot compensate for failing an important property by excelling at a minor one. The companion mechanism is the online adjustment loop of Algorithm 1: the server computes a candidate global model under the current priority order, tests it on local test sets, and if estimated accuracy has not improved, backtracks through other permutations until one improves accuracy or the least-bad candidate is kept. Together these two mechanisms let the server reweight clients every round without seeing their raw data.

What would settle it

Rerun the fully adjusted scheme on the same public benchmark, holding out a disjoint evaluation slice of each client's data that the online adjustment never sees, and count rounds to target accuracy on that slice; if the 12-versus-22 advantage over standard federated averaging disappears, the gain comes from selecting on the local test sets rather than from the criteria themselves.

Watch

Extended reading notes

Core claim

The paper's central claim is that the weight each client receives in federated averaging should encode more than local dataset size, and that a prioritized combination of three criteria—dataset size, label diversity, and local model divergence—produces a better global model than the standard size-only weighting. It further claims that the priority order among these criteria should not be fixed once by a domain expert; the server can adjust it online, choosing at each round the ordering whose candidate global model gives the highest estimated accuracy on local test sets, and backtracking when accuracy drops. On the public federated handwriting benchmark used in the experiments, the fully adjusted device-aware scheme reaches 75% accuracy for 20% of devices in 12 communication rounds, versus 22 rounds for standard federated averaging, and at high device fractions it reaches the same target in 329 mean rounds against 552.5 for the size-only baseline. The authors frame the result as evidence that aggregation weights are a meaningful design axis for federated learning.

Load-bearing premise

The online adjustment step assumes that choosing the priority ordering that maximizes accuracy on the clients' local test sets also improves the true global model; if the search instead overfits to those particular test examples, the reported gains are an artifact of peeking.

Editorial extensions

If this is right

  • If device-aware weighting is adopted, federated training can reach a target accuracy in fewer communication rounds: the best configuration needs 12 rounds to 75% accuracy for 20% of devices, versus 22 for standard federated averaging.
  • Criteria beyond dataset size—label diversity and model divergence—are worth measuring, since in isolation they already outperform the size-only baseline when a high fraction of devices must reach the target.
  • The best priority order depends on the goal: dataset size first is best for quickly serving a small fraction of devices, while model divergence first is best for a large fraction, so no single hand-picked ordering dominates.
  • Online adjustment with backtracking improves over all static permutations on average, making the scheme less sensitive to the initial priority ordering chosen by the domain expert.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same prioritized reweighting principle should transfer to other federated applications, such as recommendation or language modeling, as long as domain-specific criteria can be measured without exposing raw user data.
  • With more than a handful of criteria the exhaustive permutation search would grow factorially, so practical adoption would likely need a cheaper heuristic or a learned policy for choosing priority orders.
  • The priority order chosen online could double as an interpretable diagnostic, revealing which device properties dominate at different stages of training—a use the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes a device-aware federated learning framework. Instead of weighting local model updates only by local dataset size as in FedAvg, it computes each client's contribution from a set of normalized criteria (dataset size, label diversity, and local model divergence) using a prioritized multi-criteria aggregation operator. It further introduces an online adjustment procedure (Algorithm 1) that, at each communication round, searches over priority orderings of the criteria and selects the ordering that maximizes a weighted average of local test accuracies. Experiments on a 10% subsample of the FEMNIST dataset compare FedAvg, each criterion in isolation, all six static priority permutations, and the online-adjusted variant, reporting the number of rounds needed to reach 75% or 80% target accuracy for specified percentages of devices.

Significance. If the empirical claims were robust, the paper would be a useful practical contribution: it demonstrates how domain knowledge about clients can be incorporated into FL aggregation through a standard prioritized aggregation operator, and the idea of self-adjusting the priority order online is appealing. The static multi-criteria study is transparent in reporting all six permutations, and the use of the non-IID FEMNIST benchmark is appropriate. However, the headline advantage of the online-adjustment variant is currently not established, because the selection criterion and the reported evaluation metric are both computed on the same local test sets, and all results come from single runs without variance estimates. With a corrected evaluation protocol and repeated trials, the proposed framework could still be valuable.

major comments (2)
  1. [Section 4, Study C; Algorithm 1, lines 13-27; Table 1] The online-adjustment evaluation is circular. Algorithm 1 selects the priority ordering at each round by maximizing a weighted average of local test accuracies (lines 13-16), searching over permutations until the estimated accuracy no longer improves (lines 17-27). Table 1 then reports the number of rounds needed to reach target accuracy on those same local test sets. The reported advantage of the Final variant over FedAvg (e.g., 12 vs 22 rounds at 20% of devices for 75% accuracy) therefore reflects selection on the test data rather than an unbiased estimate of generalization. The paper should select parameters on a held-out validation split and evaluate on a separate test split, or at minimum report the current numbers as an optimistic upper bound. Notably, the Final mean over the six initializations is 20.5 rounds versus the static MCA mean of 22.5, which is a much smaller gap than the best-run 12-round result.
  2. [Section 4, Table 1; Section 3, Experimental setup] All results in Table 1 are based on single runs, and the Final variant additionally reports the best of six initializations. Without repeated seeds, multiple subsamples, or any variance estimate, differences such as 25.5 vs 27 rounds or 405 vs 552.5 rounds cannot be distinguished from noise, and reporting the best initialization as the headline is optimistic. The paper should provide means and standard deviations (or full distributions) over several independent runs and should justify any use of the best run rather than the mean.
minor comments (6)
  1. [Section 2.2, Example 1] In the second part of Example 1, the numerical computation is inconsistent with Equation (4): for the order C3, C2, C1, the weights are λ1=1, λ2=0.9, λ3=0.9·0.8=0.72, so sk = 1·0.9 + 0.9·0.8 + 0.72·0.5 = 1.98, not 1.82 as printed.
  2. [Footnote 2] Footnote 2 states '0.2×317=75 devices'; this should presumably be based on the 371 participating clients, so it should read '0.2×371≈74', and '317' appears to be a typo.
  3. [Section 4, first paragraph] The phrase 'desired target global accuracy of 75% and 80%' is confusing because the table columns report the percentage of devices reaching a target accuracy, not a single global accuracy value; please rephrase to avoid ambiguity.
  4. [Section 2.2 vs Algorithm 1] The prose after Algorithm 1 says that when no further permutations improve accuracy, 'the candidate global model which produced the least worst test accuracy is assigned to wG_{t+1}', but Algorithm 1 lines 22-24 select the ordering with the maximum accuracy. These two descriptions should be reconciled.
  5. [Section 3, Federated dataset] The statement 'we use a subsampled version (10% of total, 371 clients involved)' is ambiguous because 10% of the 3,550 FEMNIST writers would be 355, not 371; please specify exactly how the subsample was drawn.
  6. [Abstract] In the abstract, 'pushing forward the state the art in the field' should be 'pushing forward the state of the art in the field.'

Circularity Check

1 steps flagged · score 6.0 of 10

Online priority adjustment chooses the permutation by maximizing accuracy on the same local test sets used to count rounds to target accuracy, so the Final-variant improvement over FedAvg is a fitted result rather than an independent prediction.

  1. fitted input called prediction [Algorithm 1, lines 16-22; Section 4 Study C; Table 1]
    "Algorithm 1: '16: acc t+1← weighted average of acc k t+1 w.r.t. local test set size, ∀k∈A' / '22: P← priority ordering for which we get the maximum value for acc t+1' / Section 4: 'letting the server choose at each round of communication the priority ordering maximizing the accuracy (i.e, obtain the best sub-optimal accuracy).'"

    Algorithm 1 selects the priority ordering P using acc_{t+1}, which is explicitly a weighted average of accuracies measured on each device's local test set. The reported headline metric (Section 3, Table 1) is the number of rounds required for a given percentage of devices to reach a target accuracy on those same local test sets. Thus the Final row in Table 1 is not an unbiased evaluation of a learned aggregation policy; it is the value of the very objective used to choose the priority ordering at every round. The 12-round vs 22-round comparison to FedAvg is therefore statistically forced by test-set access, not an independent prediction.

full rationale

The central circularity is confined to Study C. The online adjustment procedure explicitly uses local test-set accuracies to pick, at each round, the permutation that maximizes a global accuracy estimate; the same local test sets are then used to compute the rounds-to-target-accuracy numbers in Table 1. Consequently, the Final variant's advantage, including the headline 12-round result, is not an out-of-sample prediction but a quantity obtained by repeatedly optimizing the evaluation signal. The paper does not rely on load-bearing self-citation: the prioritized aggregation operator is attributed to an external prior work, and no author-specific uniqueness theorem is invoked. Studies A and B evaluate fixed criteria and fixed priority permutations without test-set-based parameter selection, so their comparisons to FedAvg remain independently informative. The score of 6 reflects that one of the paper's main claimed improvements—the online adjustment—is partially circular because its evaluation and its selection criterion share the same data; it stops short of 8 because the algorithm still trains the underlying CNN without test labels and the target metric is not literally identical to the selection criterion. Reporting the best of six initializations without variance estimates compounds the leakage but is a statistical-reporting issue rather than a separate definitional circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a discrete free parameter (the priority ordering) that is selected using local test accuracy, and on domain assumptions about the usefulness of the criteria and the suitability of the prioritized aggregation operator. No new entities are invented.

free parameters (2)
  • Priority ordering of criteria (permutation) = Best initialization Md ≻ Ds ≻ Ld; per-round ordering selected by online search
    The priority order is not derived from theory; it is selected by the online adjustment procedure based on local test accuracy, effectively fitting the aggregation operator to the evaluation metric. The paper tests all six permutations and reports the best.
  • FEMNIST subsample fraction = 0.10
    A hand-chosen 10% subsample (371 clients) is used for computational limits; the paper gives no justification that the subsample is representative, and no seed is reported.
assumptions (4)
  • domain assumption The prioritized multi-criteria aggregation operator from da Costa Pereira et al. [6] is suitable for combining client contributions in federated learning.
    Section 2.2 adopts Eq. (4) from information retrieval without proof that the multiplicative priority weighting is appropriate for model averaging; the paper asserts its potential benefits.
  • domain assumption Label diversity and model divergence are useful predictors of a client's contribution to the global model.
    Section 2.2 states these criteria provide a clue about usefulness; no formal or empirical isolation demonstrates each criterion's causal role beyond the aggregate table.
  • ad hoc to paper Selecting the priority ordering via local test accuracy at each round does not overfit the test sets.
    Algorithm 1 lines 13-27 use LocalTestAccuracy to choose the ordering; the paper does not account for the resulting selection bias when reporting the same test accuracy as the outcome.
  • domain assumption Standard FL convergence assumptions implied by FedAvg apply when aggregation weights are replaced by non-dataset-size weights.
    The paper cites [15] for divergence but does not prove convergence of the modified weighted averaging; it relies on empirical results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Effective Device-Aware Federated Learning." pith.science (2026). https://pith.science/paper/CVJCEMDX

@misc{pith2026190807420,
  author       = {Pith},
  title        = {Pith review of: Towards Effective Device-Aware Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVJCEMDX}},
  note         = {Machine review of arXiv:1908.07420}
}
read the original abstract

With the wealth of information produced by social networks, smartphones, medical or financial applications, speculations have been raised about the sensitivity of such data in terms of users' personal privacy and data security. To address the above issues, Federated Learning (FL) has been recently proposed as a means to leave data and computational resources distributed over a large number of nodes (clients) where a central coordinating server aggregates only locally computed updates without knowing the original data. In this work, we extend the FL framework by pushing forward the state the art in the field on several dimensions: (i) unlike the original FedAvg approach relying solely on single criteria (i.e., local dataset size), a suite of domain- and client-specific criteria constitute the basis to compute each local client's contribution, (ii) the multi-criteria contribution of each device is computed in a prioritized fashion by leveraging a priority-aware aggregation operator used in the field of information retrieval, and (iii) a mechanism is proposed for online-adjustment of the aggregation operator parameters via a local search strategy with backtracking. Extensive experiments on a publicly available dataset indicate the merits of the proposed approach compared to standard FedAvg baseline.

Figures

Figures reproduced from arXiv: 1908.07420 by the authors.

Figure 1
Figure 1. An illustration of the online parameter adjustment for the aggregation operator. implementations and introduces both system and statistical rigorous metrics for understanding the quality of the FL approach. As for the metrics computation, the global model is tested on each device over the local test sets. The objective of LEAF is to capture the distribution of performance across devices by considering the 10th and 9… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 7 canonical work pages

  1. [1]

    arXiv preprint arXiv:1807.00459 (201 8)

    Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov , V.: How to backdoor federated learning. arXiv preprint arXiv:1807.00459 (201 8)

    arXiv

  2. [2]

    CoRR abs/1902.01046 (2019), http://arxiv.org/abs/1902.01046

    Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Inger man, A., Ivanov, V., Kiddon, C., Konecn´ y, J., Mazzocchi, S., McMahan, H.B., Overveldt, T.V., Petrou, D., Ramage, D., Roselander, J.: Towards federated learning at scale: System design. CoRR abs/1902.01046 (2019), http://arxiv.org/abs/1902.01046

    arXiv 2019

  3. [3]

    arXiv preprint ar Xiv:1812.01097 (2018)

    Caldas, S., Wu, P., Li, T., Koneˇ cn` y, J., McMahan, H.B., S mith, V., Talwalkar, A.: Leaf: A benchmark for federated settings. arXiv preprint ar Xiv:1812.01097 (2018)

    arXiv 2018

  4. [4]

    Annales de l’Institut Fourier 5, 131–295 (1954)

    Choquet, G.: Theory of capacities. Annales de l’Institut Fourier 5, 131–295 (1954). https://doi.org/10.5802/aif.53

    doi:10.5802/aif.53 1954

  5. [5]

    In: 2017 International Joint Confere nce on Neural Networks (IJCNN)

    Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: Emnist: Extending mnist to handwritten letters. In: 2017 International Joint Confere nce on Neural Networks (IJCNN). pp. 2921–2926. IEEE (2017)

    work page 2017

  6. [6]

    da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensi onal relevance: Pri- oritized aggregation in a personalized information retrie val setting. Inf. Pro- cess. Manage. 48(2), 340–357 (2012). https://doi.org/10.1016/j.ipm.201 1.07.001, https://doi.org/10.1016/j.ipm.2011.07.001

    work page doi:10.1016/j.ipm.201 2012

  7. [7]

    European Journal of Operational Research 89(3), 445– 456 (1996)

    Grabisch, M.: The application of fuzzy integrals in multi criteria deci- sion making. European Journal of Operational Research 89(3), 445– 456 (1996). https://doi.org/https://doi.org/10.1016/0 377-2217(95)00176-X, http://www.sciencedirect.com/science/article/pii/037722179500176X

    arXiv 1996

  8. [8]

    Fuzzy Measures and Integrals pp

    Grabisch, M., Roubens, M.: Application of the Choquet int egral in multicriteria decision making. Fuzzy Measures and Integrals pp. 348 – 374 ( 2000)

    work page 2000

Show all 18 references
  1. [9]

    CoRR abs/1511.03575 (2015), http://arxiv.org/abs/1511.03575

    Konecn´ y, J., McMahan, B., Ramage, D.: Federated optimiz ation: Dis- tributed optimization beyond the datacenter. CoRR abs/1511.03575 (2015), http://arxiv.org/abs/1511.03575

    2015 arXiv

  2. [10]

    CoRR abs/1610.02527 (2016), http://arxiv.org/abs/1610.02527

    Konecn´ y, J., McMahan, H.B., Ramage, D., Richt´ arik, P.: Federated optimization: Distributed machine learning for on-device intelligence. CoRR abs/1610.02527 (2016), http://arxiv.org/abs/1610.02527

    2016 arXiv

  3. [11]

    Proceedings of the IEEE 86(11), 2278–2324 (Nov 1998)

    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient- based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (Nov 1998). https://doi.org/10.1109/5.726791

    1998 doi

  4. [12]

    Fuzzy Sets and Systems 324, 3–19 (2017)

    Marrara, S., Pasi, G., Viviani, M.: Aggregation opera- tors in information retrieval. Fuzzy Sets and Systems 324, 3–19 (2017). https://doi.org/10.1016/j.fss.2016.12.0 18, https://doi.org/10.1016/j.fss.2016.12.018

    2017 doi

  5. [13]

    In: Proceedings of the 20th International Conference on Artific ial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdal e, FL, USA

    McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from dece ntralized data. In: Proceedings of the 20th International Conference on Artific ial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauder...

    2017

  6. [14]

    IT Professional 14(5), 53–55 (2012)

    Miller, K.W., Voas, J.M., Hurlburt, G.F.: BYOD: securit y and privacy considera- tions. IT Professional 14(5), 53–55 (2012). https://doi.org/10.1109/MITP.2012.93, https://doi.org/10.1109/MITP.2012.93

    2012 doi

  7. [15]

    arXiv preprint arXiv:1812.06127 (2018) 14

    Sahu, A.K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A ., Smith, V.: On the convergence of federated optimization in heterogeneous ne tworks. arXiv preprint arXiv:1812.06127 (2018) 14

    2018 arXiv

  8. [16]

    IEEE Trans

    Yager, R.R.: On ordered weighted averaging aggregation operators in multicrite- ria decisionmaking. IEEE Trans. Systems, Man, and Cybernet ics 18(1), 183–190 (1988). https://doi.org/10.1109/21.87068, https://doi.org/10.1109/21.87068

    1988 doi

  9. [17]

    International Journal of Intelligent Systems 11(1), 49–73 (1996)

    Yager, R.R.: Quantifier guided aggregation using owa ope rators. International Journal of Intelligent Systems 11(1), 49–73 (1996)

    1996

  10. [18]

    : Federated learning with non-iid data

    Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V. : Federated learning with non-iid data. CoRR abs/1806.00582 (2018), http://arxiv.org/abs/1806.00582

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.